Sometimes we need Adding some customized HTTP header in Jetty's Request.
Here is a way to change Request.getHeader() using java.lang.instrument and javassist.
contents of MANIFEST.MF:
command to compile and pack jar:
javac -classpath ./javassist.jar HeaderAgent.java
jar -cmf MANIFEST.MF sslagent.jar HeaderAgent.class
adding following option to your java command, then Request.getHeader() method will be customized after Request.class loaded.
-javaagent:$HOME/bin/jettySSLPlugin/sslagent.jar
git@github.com:whunmr/jettyHeaderAgent.git
2012-11-18
[emacs]go to end-of-buffer ignore whitespaces
;;we always need goto end-of-buffer, so we bind it to C-. for simple. ;;sometimes we need goto end-of-buffer, but ignore whitespaces at end-of-buffer ;;so by give end-of-buffer an advice, it'll switch between these points, when you ;;continue typing C-. (defadvice end-of-buffer (around move-backward-skip-whitespace) "Never use tabs for alignment." (let ((point-before-move (point))) ad-do-it (if (and (eq point-before-move (point)) (eq last-command this-command)) (progn (search-backward-regexp "[^\n\s-]\\W*") (forward-char))) )) (ad-activate 'end-of-buffer) (define-key my-keys-minor-mode-map (kbd "C-.") 'end-of-buffer)
-->
2012-11-09
[emacs] compare with clipboard like IntelliJ IDEA
(defun compare-with-clipboard () "compare current buffer with contents in clipboard" (interactive) (let ((clipboard-buffer (get-buffer-create "*clipbord*")) (orig-buffer (current-buffer))) (switch-to-buffer clipboard-buffer) (delete-region (point-min) (point-max)) (yank) (ediff-buffers clipboard-buffer orig-buffer)))
-->
2012-11-06
Monitor process file operations by windbg command
* Monitor windows process activities by set breakpoint on API, and print out the parameter contents. 0:000> bp kernel32!CreateFileW ".echo ---------------------------------------;kL;du poi(@esp+4);gu;.echo =======;r eax;g" 0:000> g ModLoad: 62c20000 62c29000 C:/WINDOWS/system32/LPK.DLL ModLoad: 77180000 77283000 C:/WINDOWS/WinSxS/x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03/comctl32.dll --------------------------------------- ChildEBP RetAddr 0012e374 7c814d65 kernel32!CreateFileW 0012e5dc 7c801d3a kernel32!BasepLoadLibraryAsDataFile+0x125 0012e640 7c8171dd kernel32!LoadLibraryExW+0x178 0012e66c 7c81715d kernel32!BasepSxsFindSuitableManifestResourceFor+0x51 0012e96c 7720b80d kernel32!CreateActCtxW+0x69e 0012eba4 7720b83f comctl32_77180000!SHFusionInitializeIDCC+0x83 0012ebb8 7720b857 comctl32_77180000!SHFusionInitializeID+0x12 0012ebc8 771841a9 comctl32_77180000!SHFusionInitialize+0xf 0012ebdc 77184267 comctl32_77180000!_ProcessAttach+0x32 0012ebe8 7c9211a7 comctl32_77180000!LibMain+0x21 0012ec08 7c93cbab ntdll!LdrpCallInitRoutine+0x14 0012ed10 7c936178 ntdll!LdrpRunInitializeRoutines+0x344 0012efbc 7c9362da ntdll!LdrpLoadDll+0x3e5 0012f264 7c801bb9 ntdll!LdrLoadDll+0x230 0012f2cc 7c80ae5c kernel32!LoadLibraryExW+0x18e 0012f2e0 77f5b1a3 kernel32!LoadLibraryW+0x11 0012f504 766a1110 SHLWAPI!LoadLibraryWrapW+0x51 0012f53c 766a10af WININET!SHFusionLoadLibrary+0x29 0012f548 766a107d WININET!DelayLoadCC+0x15 0012f77c 766a0ff7 WININET!SHFusionInitializeIDCC+0x92 0012e3d0 "C:/WINDOWS/WindowsShell.Manifest" 0012e410 "" ======= eax=00000794 --------------------------------------- ChildEBP RetAddr 0012f8cc 7c801a4f kernel32!CreateFileW 0012f8f0 76d357ff kernel32!CreateFileA+0x30 0012f954 76d3570a iphlpapi!OpenIPDriver+0x115 0012f99c 76d35454 iphlpapi!OpenTCPDriver+0xee 0012f9d0 76d35351 iphlpapi!DllMain+0x157 0012f9f0 7c9211a7 iphlpapi!_DllMainCRTStartup+0x52 0012fa10 7c93cbab ntdll!LdrpCallInitRoutine+0x14 0012fb18 7c94173e ntdll!LdrpRunInitializeRoutines+0x344 0012fc94 7c941639 ntdll!LdrpInitializeProcess+0x1131 0012fd1c 7c92eac7 ntdll!_LdrpInitialize+0x183 00000000 00000000 ntdll!KiUserApcDispatcher+0x7 7ffdfc00 "//./Ip" ======= eax=00000780 --------------------------------------- *** ERROR: Symbol file could not be found. Defaulted to export symbols for C:/Program Files/Tencent/QQ2009/Bin/KernelUtil.dll - ChildEBP RetAddr 0012fc34 0044a69c kernel32!CreateFileW *** ERROR: Module load completed but symbols could not be loaded for QQ.exe WARNING: Stack unwind information not available. Following frames may be wrong. 0012fca8 004029bd KernelUtil!Version::Init+0x8c 0012ff08 004027d9 QQ+0x29bd 0012ff28 00402635 QQ+0x27d9 0012ffc0 7c816fd7 QQ+0x2635 0012fff0 00000000 kernel32!BaseProcessStart+0x23 00c8eca8 "C:/Program Files/Tencent/QQ2009/" 00c8ece8 "Bin/vi.dat" ======= eax=00000720 --------------------------------------- ChildEBP RetAddr 0012eab4 0044a2ad kernel32!CreateFileW WARNING: Stack unwind information not available. Following frames may be wrong. 0012fbb4 0044a4ea KernelUtil!Util::URL::OpenUrlWithTT+0x1cd 0012fc4c 0044a6fc KernelUtil!Version::GetBuildVer+0xca 0012fca8 004029bd KernelUtil!Version::Init+0xec 0012ff08 004027d9 QQ+0x29bd 0012ff28 00402635 QQ+0x27d9 0012ffc0 7c816fd7 QQ+0x2635 0012fff0 00000000 kernel32!BaseProcessStart+0x23 00c8ede0 "C:/Program Files/Tencent/QQ2009/" 00c8ee20 "Bin/QQ.exe" ======= eax=00000720
-->
2012-11-05
Start from helloworld.asm
* asm ** gdb::: resource :: stop at main :: view register ::info registers show single reg ::print/(d|t|x) $eax show memory value ::x/<num>(c|d|x)(b|h|w) x/42cb &output (gdb) x/s $edi+28 0x600104 <output+28>: "GenuineIntel'\n" ** common Code of helloworld.asm:: SECTION .data EatMsg: db "hello world", 10 EatLen: equ $-EatMsg SECTION .bss SECTION .text global _start _start: nop mov eax, 4 mov ebx, 1 mov ecx, EatMsg mov edx, EatLen int 80H mov eax, 1 mov ebx, 0 int 80H Command to build:: [vagrant@vagrant-centos-6-64 vagrant] helloworld.asm helloworld.o [vagrant@vagrant-centos-6-64 vagrant] helloworld helloworld.asm helloworld.o [vagrant@vagrant-centos-6-64 vagrant] hello world dump all sections :: 08048080 <_start>: ||break *_start+1 then %epi will contains 0x08048080 8048080: 90 nop 8048081: b8 04 00 00 00 mov $0x4,%eax 8048086: bb 01 00 00 00 mov $0x1,%ebx gcc output :: gcc -E :: compile wit debug info :: dump code with assembly :: int foo() { 400684: 55 push %rbp 400685: 48 89 e5 mov %rsp,%rbp 400688: 48 83 ec 10 sub $0x10,%rsp 40068c: e8 a7 fe ff ff callq 400538 <mcount@plt> ||<--insert by -gp for gprof #include <stdio.h> int main() { 400504: 55 push %rbp 400505: 48 89 e5 mov %rsp,%rbp printf("Hello, World!\n"); 400508: bf 18 06 40 00 mov $0x400618,%edi 40050d: e8 de fe ff ff callq 4003f0 <puts@plt> exit(0); 400512: bf 00 00 00 00 mov $0x0,%edi 400517: e8 e4 fe ff ff callq 400400 <exit@plt> dump out dynamic symbol table entries:: ctest: file format elf64-x86-64 DYNAMIC SYMBOL TABLE: 0000000000000000 w D *UND* 0000000000000000 __gmon_start__ 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 puts 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 exit 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __libc_start_main dump dynamic entries:: Dynamic section at offset 0x2b0 contains 15 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] show used share libs:: libc.so.6 => /lib64/libc.so.6 (0x00007f0e83eb4000) /lib/ld64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f0e8424f000) mov:: movl(long word) movw(16-bit word) movb(byte) | movl %eax, %ebx | movw %ax %bx | movb %al %bh conditional move:: movl value, %ecx | cmp %ebx, %ecx | cmova %ebx, %ecx XCHG::exchange value between two general purpos registers, or betwen a register and a memory location. when one of the operands is a memory location, the processor's LOCK signal is automatically asserted, to turns the instruction into an atomic instruction in multiple processor environment. CMPXCHG:: cmpxchg src dest | compare dest with EAX,AX,or AL. if equal, load src to dest. otherwise load dest to EAX, AX, or AL. unconditional branches::jump calls interrupts call: 1) push the $eip, modifies the %eip to point to called function address 2) call function and ret 3) pop %eip to continue asm function template:: function_label: pushl %ebp movl %esp, %ebp <normal function code> movl %ebp, %esp popl %ebp software interrupts::are provided by OS to enable app to tap info function into the OS. (system calls). ** resources http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
-->
2012-11-02
What Emacs can do?
code alignment (gif video)
column editing (gif video)
WEIBO, git, git blame, rainbow (gif video)
dynamic selection and Code browsing (gif video)
org-mode + googledriver to manage your notes in one file (gif video)
weibo and music
Task management in TDD
2012-10-23
Reading notes of Learning the vi and Vim Editors
* Notes of Learning the vi and Vim Editorsrepeat last command :: . Movements :: w W b B e E ( ) { } [[ 0 ^ n| $ gg G 200G move to char :: fx Fx tx Tx (; continue forward and ' for backward) move on screen :: H M L 2H Actions :: d c y append at line end :: A insert at line begin :: I delete char :: x X open line up and down :: o O paste up and line :: p P delete line and insert :: S //same as ddi or cc (number prefix)write 50 * :: 50i* set-mark in vi :: mx (x for any letter) goto mark :: `x delete line :: dd delete from cursor to end :: D undo :: u U redo :: ctrl+R center current line :: Z. paste from named delete buffer :: "3p yank to named buffera :: "a4yy append to named buffer a :: "A4yy paste from named buffer :: "ap paste :: p P single char :: x r search :: / ? n N line-matches-regex :: g/pattern/p line-non-matches-regex :: g!/pattern replace once :: s/old/new emit another replace in command mode :: & replace all occurs in line :: s/old/new/g replace all occurs in file :: %s/old/new/g (% stands for all lines, like 1,% prefix to s command 1,%s/old/new/g) replace with confirm :: s/old/new/gc only do replace in line matches patterns :: :g/pattern/s/old/new/g add () to lines :: :%s/.*/(&)/ (& stands for whole match line) replace with regex :: :%s/\(this\) or \(that\)/\2 or \1/g replace Fortran to FORTRAN :: %s/Fortran/\U&/ (also has \L for lowercase) save buffer before replace :: :w revert buffer to last saved state :: e! edit another file foo :: :e foo switch files in vi :: :n :rewind :last line info :: ctrl+G append to word end :: ea delete to file end :: dG change to line end :: c$ C goto line 100 :: 100G toggle line number :: :set nu :set nonu split window :: :split :vsplit move between windows :: ^wj ^wk ^wh ^wl
2012-10-10
[emacs] copy until char (like zap-to-char)
Today, When we doing a story with a piece of XML like this: <suburb>S-123456789</suburb> I feel not convenient to copy out the S-123456789 So I wrote a new copy-without-selection function copy-to-char function (like zap-to-char):
(defun copy-to-char (arg char) "Copy until char like zap-to-char" (interactive "p\ncZap to char: ") (let ((start (point))) (save-excursion (copy-region-as-kill start (progn (search-forward (char-to-string char) nil nil arg) (backward-char) (point))) ) ) ) (global-set-key (kbd "C-c c") 'copy-to-char)
2012-09-20
reading notes of Programming Ruby 2nd
#!/usr/bin/ruby -w # -*- coding: utf-8 -*- animals = %w{ant bee cat dog elk} def say_hello (name) "Hello, #{name.capitalize}" end puts say_hello('ww') $globalV = "hello" @instanceV = "instanceV" @@classV = "ww" puts "#$globalV, #@instanceV #@@classV" hash_example = { 'a' => 'A', 'b' => 'B', 'c' => 'C' } puts hash_example['c'] hash_default_value_zero = Hash.new(0) puts hash_default_value_zero['key-do-not-exists'] == 0 testflag = true puts "use simple if statement, if possible" if testflag puts "regex demo: if perl match /perl|python/" if "perl" =~ /perl|python/ aline = 'Perl and Python' aline.sub(/Perl/, 'Ruby') # replace first 'Perl' with 'Ruby' aline.gsub(/Python/, 'Ruby') # replace every 'Python' with 'Ruby' aline.gsub(/Perl|Python/, 'Ruby') animals.each { |animal| puts animal } 5.times { print "*" } 3.upto(6) {|i| print i } ('a'..'e').each {|char| print char } class Song attr_reader :name, :artist, :duration, :attr_foo attr_writer :name, :artist, :duration protected :attr_foo def initialize(name, artist, duration) @name = name @artist = artist @duration = duration end def to_s "Song: #@name, #@artist, #@duration " end end class SubSong < Song CONSTANTS_START_WITH_UPPERCASE_LETTER = 300 def initialize(name, artist, duration, lyrics) super(name, artist, duration) @lyrics = lyrics end def SubSong.staticMethod "static method" end def [] (seconds) #index method, e.g: SubSong[22] seconds end def play end def pause end def inspect '#' * 10 end end s = SubSong.new("a", "b", "c", "lyrics") puts puts s.duration puts s.name puts SubSong.staticMethod class SingletonLogger private_class_method :new @@logger = nil def SingletonLogger.get_instance @@logger = new unless @@logger @@logger end def self.static_method_form_b end class << self def static_method_form_c end end def public_method_a end public :public_method_a protected def protected_method_a end private def private_method_a end end i = SingletonLogger.get_instance puts i.object_id == SingletonLogger.get_instance.object_id puts i.class j = i.dup #duplicate a new object puts j.class j.freeze #freeze an object array = Array.new array[0] = "xxx" array[1] = "yyy" array[2] = "zzz" puts array[0,2] puts array[100] == nil #a=[1,3,5,7,9] → [1,3,5,7,9] #a[2, 2] = ’cat’ → [1, 3, "cat", 9] #a[2, 0] = ’dog’ → [1, 3, "dog", "cat", 9] #a[1,1]=[9,8,7] → [1,9,8,7,"dog","cat",9] #a[0..3] = [] → ["dog", "cat", 9] #a[5..6] = 99, 98 → ["dog", "cat", 9, nil, nil, 99, 98] require 'test/unit' class TestSong < Test::Unit::TestCase def test_method h = { 'a' => 'A', 'b' => 'B', 'c' => 'C'} assert_equal(h['c'], 'C') end end def two_times yield yield end two_times { puts "Hello"} a = [1, 'cat', 3.14] #array with three elements a[2] = nil puts a.find {|element| element == 1} puts [1,3,5,7,9].find{|v|v*v>30} puts [1,3,5,7,9].each {|i| puts i} puts ['S', 'V'].collect {|x| x.succ} puts defined?(not_defined_variable) == nil puts [1,2,3,4].inject {|p, e| p*e} class File def File.with_file_open(*args) result = f = File.open(*args) if block_given? result = yield f f.close end f end end song = SubSong.new("a", "b", "c", "d") p s class SongButton def initialize(label, &action) @label = label @action = action end def button_pressed @action.call(self) end end start_button = SongButton.new("Start") { song.start } def n_times(thing) return lambda { |n| thing *n } end proc_3_times = n_times("hello") puts proc_3_times.call(3) num = 81 5.times do puts "#{num.class}: #{num}" num *= num end 99.downto(96) { |i| print i, " " } 50.step(80,5) { |i| print i, " " } puts "#{'hello ' * 3}" puts "now is #{ def the(a) 'the ' + a end the('time') } for all.." p (1..10).to_a p ('bar'..'bat').to_a digits = 0..9 puts digits.include?(5) puts 1 <=> 2 puts 1 <=> 1 puts (1..10) === 5 puts (1..10) === 12 #102 More about methods #method name instance_of? chop chop! name= def method_without_parameters_do_not_need_parenthesis puts ' ' end def method_parameter_can_have_default_value(input="hello") puts input end method_parameter_can_have_default_value def varargs(arg1, *rest) "Got #{arg1} and #{rest.join(', ')}" end puts varargs("first variable", "and", "more", "params") #if block_given? #if omit the receiver, it defaults to self. #self.frozen? is same as frozen? #self.id is same as id #Modules module Module_Can_do_mixin PI = 3.14 def morsecode end end module Debug def who_am_i "#{self.class.name}" end end class Photo include Debug end require 'open-uri' open('http://www.sina.com.cn') do |f| puts f.read.scan(/<img src="(.*?)"/m).uniq end #Threads and Processes #P377 to be continue...
2012-09-19
copy to end of line from current point
As an Emacs user, maybe you already known CopyWithoutSelection
But currently without copy to line end from current point.
Following is the my implementation, and bind to the shortcut "C-c e"
init.el
But currently without copy to line end from current point.
Following is the my implementation, and bind to the shortcut "C-c e"
(defun copy-to-line-end (&optional arg) "Save point to current line end into Kill-Ring without mark (or kill and yank)" (interactive "P") (let ((start (point))) (save-excursion (end-of-line) (copy-region-as-kill start (point)) ) ) ) (global-set-key (kbd "C-c e") 'copy-to-line-end)
2012-09-03
Using Jenkins' Script Console to Download Files
* Using Jenkins Script Console to Download Files Scenario:: ServerA has a file test.t want to copy to ServerB, But don't have login user on ServerB. And ServerB running a Jenkins server, we can access. Solutions:: On ServerA: 1. goto folder of test.t 2. execute python -m SimpleHTTPServer 19:41 ~ Serving HTTP on 0.0.0.0 port 8000 ... On Jenkins:: 1. execute following code in Jenkins's Script Console.String cmd ="curl <ip_of_server_A>:8000/test.t -o test.t"; Runtime run = Runtime.getRuntime(); Process pr = run.exec(cmd); pr.waitFor(); BufferedReader buf = new BufferedReader(new InputStreamReader(pr.getInputStream())); String line = ""; while ((line=buf.readLine())!=null) { println line; }Then, test.t will located in Home folder of user who running the Jenkins. 2. Surely, any Groovy Script can execute here.
2012-08-26
handy commands for development
COMMAND:: Watch something to happen :: Find all handle of App :: Find files opened by App :: Find which App is using specified socket port 8080 :: Monitor file changes :: Grep except something :: Monitor CPU balance :: Monitor system performance :: Using python to start a HTTP Server to share files :: call command with ! followed by first letter of recent cmd :: pushd and popd to navigation in dir :: - console:: Ctrl + u Erase the current line. Ctrl + k Delete the line from the position of the cursor to the end of the line. Ctrl + w Delete the word before the cursor. Ctrl + r incremental search previously called commands, continue type Ctrl+ r to search next Ctrl + a Move to line beginning Ctrl + e Move to line end Ctrl + d Delete char under cursor Ctrl + _ Undo ESC b Move backward one word ESC f Move forward one word ESC d Delete word right ESC backspace Delete word left let mysql log the queries:: mysql -uroot -pYOUR-PASSWORD -e "SET GLOBAL log_output = 'FILE'; Set GLOBAL general_log_file = '/tmp/mysql.log'; SET GLOBAL general_log = 'ON';" mac os:: - To move files in Finder first press Command+C to copy the selected files, then press Command+Option+V to move the copied files to the current folder. - ctrl+F2 to focus on menu bar - command+shift+/ activate Help and search menu items with name - pbcopy - pbpaste - Quicksilver open directory in iTerm - cmd+D split iTerm window
-->
2012-08-25
Favorite emacs key-bindings
(global-set-key "\M-n" 'next-buffer)
(global-set-key "\M-p" 'previous-buffer)
(global-set-key (kbd "C-c w") (quote copy-word))
(global-set-key (kbd "C-c l") (quote copy-line))
(global-set-key (kbd "C-c p") (quote copy-paragraph))
(global-set-key (kbd "C-c s") (quote thing-copy-string-to-mark))
(global-set-key (kbd "C-c a") (quote thing-copy-parenthesis-to-mark))
(global-set-key "\C-o" 'other-window)
(global-set-key "\M-o" 'other-window)
(defvar my-keys-minor-mode-map (make-keymap) "my-keys-minor-mode keymap.")
(define-key my-keys-minor-mode-map (kbd "C-,") 'beginning-of-buffer)
(define-key my-keys-minor-mode-map (kbd "C-.") 'end-of-buffer)
(define-minor-mode my-keys-minor-mode
"A minor mode so that my key settings override annoying major modes."
t " my-keys" 'my-keys-minor-mode-map)
(my-keys-minor-mode 1)(global-set-key (kbd "C-M-'") 'er/expand-region)
(define-key global-map (kbd "C-c SPC") 'ace-jump-mode)
(define-key global-map (kbd "C-;") 'ace-jump-char-mode)
(define-key global-map (kbd "C-'") 'ace-jump-line-mode)
(require 'mark-more-like-this)
(global-set-key (kbd "C-<") 'mark-previous-like-this)
(key-chord-define-global "zz" 'mark-previous-like-this)
(global-set-key (kbd "C->") 'mark-next-like-this)
(key-chord-define-global "??" 'mark-next-like-this)
(global-set-key (kbd "C-M-m") 'mark-more-like-this)
(global-set-key (kbd "C-*") 'mark-all-like-this)
(key-chord-define-global ",," 'mark-all-like-this)
(key-chord-define-global "zk" 'zap-to-char)
(key-chord-define-global ";w" 'kill-region)
;;accumulate-kill
(defun append-kill-line (&optional arg)
"Append kill-line to current kill buffer, prefix arg kills from beginning of line."
(interactive "P")
(append-next-kill)
(kill-line arg)
)
(define-key global-map "\C-x\C-m" 'append-kill-line)
2012-08-24
mongodb-mapreduce and highcharts
During company's hackday, we analyze some bigdata in mongodb.
We want to find out whether XXXXX price's short term and long term Moving averages can predicting the future price changing trends.
BACKGROUND:
http://www.hardrightedge.com/tactics/indicators/movingaverages.htm
http://winninginvestor.quickanddirtytips.com/The-Top-5-Chart.aspx
THE SCRIPT TO MAP-REDUCE DATA IS:
m = function () {
if (this.landsize > 0) {
seq_key = NumberInt(this.seq / 100);
if (this.valuation_range_high > 0 && this.valuation_range_low > 0) {
emit(seq_key, {seq:seq_key, avg_price:0, price:(this.valuation_range_high + this.valuation_range_low) / this.landsize, count:1});
}
}
}
r = function (key, values) {
r = {seq:key, avg_price:0, price:0, count:0};
values.forEach(function (v) {r.price += v.price;r.count += v.count;});
return r;
}
function finalizef(key, value) {
if (value.count > 0) {
value.avg_price = value.price / value.count / 2;
}
return value;
}
s = db.rpdata.mapReduce(m, r, {finalize:finalizef, out : "price"})
result::
{
"result" : "price",
"timeMillis" : 730604,
"counts" : {
"input" : 19477443,
"emit" : 7345947,
"reduce" : 26799,
"output" : 19468
},
"ok" : 1,
}
THE PYTHON SCRIPT TO LOAD DATA FROM MONGODB:
from pymongo import Connection
import pymongo
...
def get_price_from_db(self):
connection = Connection()
db = connection.bigdata
price_records = db.price
avg_prices = []
for record in price_records.find():
avg_prices.append(record['value']['avg_price'])
mean_value = mean(avg_prices)
std_value = std(avg_prices)
normalized_prices = []
for price in avg_prices:
//using z-score to strip strange values
if abs((price - mean_value) / std_value) < 1.0 :
normalized_prices.append(price)
Code on github
REFERENCES: moving_averages http://en.wikipedia.org/wiki/Moving_average http://en.wikipedia.org/wiki/Z-score http://www.mongodb.org/display/DOCS/MapReduce http://highcharts.com/demo/ http://docs.python.org/library/basehttpserver.html http://www.tanzilli.com/python_httpserver
2012-08-17
Enhance htmlize.el , now can export org-link.
Several Days ago, I found a problem when write blog using org-mode.
The 'html-region command in htmlize.el, just generate a CSS style for org-link, not a real link.
So I went to great stackoverflow.com:
http://stackoverflow.com/questions/11937649/in-emacs-how-to-export-links-to-a-clickable-link-when-htmlize-emacs-buffer
----------------------------------------------------------------------------------------------------------------
@Andreas gave a great hint, so I tried to read the code of org-mode,
after write some elisp function, finally it worked.
The new htmlize.el was shared on github.com now:
https://github.com/whunmr/dotemacs/blob/master/site-lisp/htmlize.el
----------------------------------------------------------------------------------------------------------------
And this the main code:
The 'html-region command in htmlize.el, just generate a CSS style for org-link, not a real link.
So I went to great stackoverflow.com:
http://stackoverflow.com/questions/11937649/in-emacs-how-to-export-links-to-a-clickable-link-when-htmlize-emacs-buffer
BACKGROUND
- I using great htmlize.el to export my org-mode buffer contents with font hi-lock.
- Emacs org-mode has a Link format.
PROBLEM
For Example, here is a org-mode file with contents:
[[http://hunmr.blogspot.com][blog]]
When I Using Htmlize.el to htmlize buffer to HTML contents, The link was missing. produces HTML like:
<span style="hyperlinkFOOBAR">blog</span>
EXPECTED
I expected it produces clickable link like:
<a style="hyperlinkFOOBAR" href="http://hunmr.blogspot.com">blog</a>
QUESTION
EDIT1 The org-export-as-html can export link, but can not create CSS for the Hi-locks.
- Do you know other ways to to export org-mode links to HTML?
- To read the real link in org-mode buffer using elisp, how to do that? read text property?
----------------------------------------------------------------------------------------------------------------
@Andreas gave a great hint, so I tried to read the code of org-mode,
after write some elisp function, finally it worked.
The new htmlize.el was shared on github.com now:
https://github.com/whunmr/dotemacs/blob/master/site-lisp/htmlize.el
----------------------------------------------------------------------------------------------------------------
And this the main code:
(defun expand-org-link (&optional buffer)
"Change [[url][shortname]] to [[url] [shortname]] by adding a space between url and shortname"
(goto-char (point-min))
(while (re-search-forward "\\[\\[\\([^][]+\\)\\]\\(\\[\\([^][]+\\)\\]\\)?\\]"
nil t)
(let ((url (match-string 1))
(link-text (match-string 3)))
(delete-region (match-beginning 0) (match-end 0))
(insert "[[" url "] [" link-text "]]"))))
(defun shrink-org-link (&optional buffer)
"Change [[url] [shortname]] to [[url][shortname]], remove the space between url and shortname"
(goto-char (point-min))
(while (re-search-forward "\\[\\[\\([^][]+\\)\\] \\(\\[\\([^][]+\\)\\]\\)?\\]"
nil t)
(let ((url (match-string 1))
(link-text (match-string 3)))
(delete-region (match-beginning 0) (match-end 0))
(insert "[[" url "][" link-text "]]"))))
(defun transform-org-link ()
"transform htmlized to "
(goto-char (point-min))
(while (re-search-forward "\\[\\[]+\\)>\\([^][]+\\)\\] \\[\\([^][]+\\)\\]\\]"
nil t)
(let ((style (match-string 1))
(url (match-string 2))
(link-text (match-string 3)))
(delete-region (match-beginning 0) (match-end 0))
(insert "" link-text ""))))2012-08-12
Java Performance Tuning
* What I found in Performance Tuning
** BETTER MACHINE
- "we need more CPU core. we need more memory, currently 96G is not enough on our server. we need SSD"
** BETTER ALGORITHM - using Array instead of HashMap || - ASN.1 parse || - There is a system, Every component has a uuid, but that's NOT efficient to dispatch calls. || So we assigned each component a numeric ID for fast dispatch.
** CHOOSE PROPER INITIAL SIZE
- StringBuilder
- HashMap && ConcurrentHashMap && ArrayList ...
- YOUNG & OLD generation's size
+ GC in JAVA
GC for Eden Space:
GC for Old Generation:
+ HotSpot VM options:
-Xmn
-Xms256m -Xmx512m ||Set initial and maximum to equal, to avoid using CPU cycle to grow heap size.
-XX:NewRatio=n //Ratio of new/old generation sizes. The default value is 2.
-XX:SurvivorRatio=n //Ratio of eden/survivor spacesize. The default value is 8.
-XX:MaxTenuringThreshold=n
-XX:PermSize -XX:MaxPermSize
-XX:UseConcMarkSweepGC
-XX:ParallelGCThreads=n
+ If most of the application's data are short lived, you should expand size of YOUNG generation.
otherwise, expand the size of OLD generation.
find more GC options of HotSpot VM
+ About java GC GC Collectors ‐XX:+UseConcMarkSweepGC
+ SoftReference, WeakReference + Use case of PhantomReference + Apple has a new Compile-time tech to avoid performance problem caused by GC: Automatic Reference Counting
** LOW DOWN THE MEMORY CONSUMPTION
- String's memory structure in JVM
- structure of Object in JVM
+ a normal object requires 8 bytes of "housekeeping" space;
OpenJDK6: src/hotspot/src/share/vm/oops/markOop.hpp
// 32 bits:
// --------
// hash:25 ------------>| age:4 biased_lock:1 lock:2 (normal object)
// JavaThread*:23 epoch:2 age:4 biased_lock:1 lock:2 (biased object)
// size:32 ------------------------------------------>| (CMS free block)
// PromotedObject*:29 ---------->| promo_bits:3 ----->| (CMS promoted object)
+ arrays require 12 bytes (the same as a normal object, plus 4 bytes for the array length).
- structure of String Object public final class String 8 |String's object header 4 |private final java.lang.Object value;--------->, 4 |private final int offset; | 4 |private final int count; | 4 |private int hash; | | 8 |Char Array's object header |<-----------------' 4 |array length | N*2 |bytes of N characters | P |bytes of padding to 8n | so, empty string will using 40 bytes. In 64bits JVM, Object header will use 16bytes. ||Using AnsiString, with byte[] rather than char[]
- string's substring implementation
||why we need create a new String after substring. //new String(str.substring(5, 4));
- StringBuilder/StringBuffer use less memory.
- ||If not concern string contents in computation and caching, you can create your own String
|| constant pool(HashMap) to convert frequently used String to Integer value.
|| when the string is never use anymore, convert it back.
- memory compress
- snapp-java
- Out of heap memory #find more
- BigMemory
- Story about not use AtomicInteger
- has better performance than synchronize adding (interlockedincrement __sync_fetch_and_add)
- use more memory
- IN THAT STORY: can be avoided through hash re-Dispatch
- AtomicReference
Java volatile reference vs. AtomicReference ( compareAndSet() used in Queue implementation)
** TOOLS *** jps use jps to find correct jvm process IdC:\Documents and Settings\Administrator>jps -lmVv 3672 org.nasa.marsrovers.Main -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:4176 -Dfile.encoding=UTF-8 -Xbootclasspath:C:\Program Files\Java\jre6\lib\resources.jar;C:\Program Files\Java\jre6\lib\rt.jar;C:\Program Files\Java\jre6\lib\jsse.jar;C:\Program Files\Java\jre6\lib\jce.jar;C:\Program Files\Java\jre6\lib\charsets.jar 3380 sun.tools.jps.Jps -lmVv -Denv.class.path=.;C:\Program Files\Java\jdk1.6.0_22\jre\lib;C:\Program Files\Java\jdk1.6.0_22\lib; -Dapplication.home=C:\Program Files\Java\jdk1.6.0_22 -Xms8m 552 -Dosgi.requiredJavaVersion=1.5 -Xms40m -Xmx384m -XX:MaxPermSize=256m
*** jstack ||Find out current stack trace. ||Thread name will show in stack trace, that's why we need set name for thread and threadPool. + look stack pattern find bug ||DeadLock in webserver We reproduced three days. If we grab a thread dump before restart server, This bug will be super easy to find. same as .NET and C++. before restart server, first thing we need to do is collect more information (dump, memory usage, CPU usage) ||HashMap infinite loop ||Most thread blocked on LOG message queue's put()
$ jstack -l 36722012-08-12 22:48:17 Full thread dump Java HotSpot(TM) Client VM (17.1-b03 mixed mode): "Low Memory Detector" daemon prio=6 tid=0x16bb2800 nid=0x1314 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x16baf400 nid=0xbd4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Command Reader" daemon prio=6 tid=0x16bad000 nid=0xdc4 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Event Helper Thread" daemon prio=6 tid=0x16bab000 nid=0x16c8 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Transport Listener: dt_socket" daemon prio=6 tid=0x16ba8c00 nid=0x11d4 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x16b99000 nid=0x168c waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x16bb3400 nid=0x1320 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x16b85000 nid=0x1678 in Object.wait() [0x16cff000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x029d0b28> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x029d0b28> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x16b83c00 nid=0x13ec in Object.wait() [0x16caf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x029d0a28> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x029d0a28> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "main" prio=6 tid=0x00847000 nid=0x1464 runnable [0x0091f000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(Unknown Source) at java.io.BufferedInputStream.read1(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x029e19f0> (a java.io.BufferedInputStream) at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) at sun.nio.cs.StreamDecoder.implRead(Unknown Source) at sun.nio.cs.StreamDecoder.read(Unknown Source) - locked <0x02b63a00> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(Unknown Source) at java.io.BufferedReader.fill(Unknown Source) at java.io.BufferedReader.readLine(Unknown Source) - locked <0x02b63a00> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(Unknown Source) at org.nasa.marsrovers.simulator.Simulator.startUp(Simulator.java:53) at org.nasa.marsrovers.Main.main(Main.java:50) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x16b81400 nid=0xef0 runnable "VM Periodic Task Thread" prio=10 tid=0x16bc9000 nid=0x304 waiting on condition JNI global references: 1508Attaching to process ID 623, please wait... Debugger attached successfully. Server compiler detected. JVM version is 20.6-b01-414 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 838860800 (800.0MB) NewSize = 21757952 (20.75MB) MaxNewSize = 174456832 (166.375MB) OldSize = 65404928 (62.375MB) NewRatio = 7 SurvivorRatio = 8 PermSize = 21757952 (20.75MB) MaxPermSize = 367001600 (350.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 19595264 (18.6875MB) used = 16274240 (15.52032470703125MB) free = 3321024 (3.16717529296875MB) 83.0519047867893% used Eden Space: capacity = 17432576 (16.625MB) used = 15573992 (14.852516174316406MB) free = 1858584 (1.7724838256835938MB) 89.33844315378289% used From Space: capacity = 2162688 (2.0625MB) used = 700248 (0.6678085327148438MB) free = 1462440 (1.3946914672851562MB) 32.37859552556818% used To Space: capacity = 2162688 (2.0625MB) used = 0 (0.0MB) free = 2162688 (2.0625MB) 0.0% used concurrent mark-sweep generation: capacity = 154337280 (147.1875MB) used = 120364648 (114.7886734008789MB) free = 33972632 (32.398826599121094MB) 77.98805836153132% used Perm Generation: capacity = 176562176 (168.3828125MB) used = 134844944 (128.59815979003906MB) free = 41717232 (39.78465270996094MB) 76.37249781062961% used
**** show memory usage according object type 09:02 ~ $ jmap -histo 623 | head -n 20num #instances #bytes class name ---------------------------------------------- 1: 57975 34551008 [B 2: 304951 33637464 [C 3: 166951 23748152 <constMethodKlass> 4: 166951 22721672 <methodKlass> 5: 21247 21762832 <constantPoolKlass> 6: 279214 18444400 <symbolKlass> 7: 21247 17749960 <instanceKlassKlass> 8: 20092 13129152 <constantPoolCacheKlass> 9: 304840 9754880 java.lang.String 10: 53366 4497624 [Ljava.lang.Object; 11: 15038 4360504 [I 12: 122304 3913728 java.util.HashMap$Entry 13: 22426 2332304 java.lang.Class 14: 4146 2094728 <methodDataKlass> 15: 32534 1717344 [S 16: 12637 1676168 [Ljava.util.HashMap$Entry; 17: 35117 1544960 [[I
**** get a JVM heap dump $ jmap -dump:file=/tmp/demo.map 91440 Dumping heap to /private/tmp/demo.map ... Heap dump file created or -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=C:/temp/oom.hprof **** analyze heap dump + $ jhat /private/tmp/demo.map Reading from /private/tmp/demo.map... Started HTTP server on port 7000
+ Like !dumpheap -type Exception in windbg+sos, you can use OQL to find out
more useful stuff in the dumped heap, like:
select file.path.toString() from java.io.File file
**** using eclipse MAT http://www.eclipse.org/mat to analyze the dump file.
- articles about MAT
**** Monitor GC status of JVM - jstat $ jstat -gcutil 21891 250 7 S0 S1 E O P YGC YGCT FGC FGCT GCT 12.44 0.00 27.20 9.49 96.70 78 0.176 5 0.495 0.672 12.44 0.00 62.16 9.49 96.70 78 0.176 5 0.495 0.672 12.44 0.00 83.97 9.49 96.70 78 0.176 5 0.495 0.672 0.00 7.74 0.00 9.51 96.70 79 0.177 5 0.495 0.673 0.00 7.74 23.37 9.51 96.70 79 0.177 5 0.495 0.673 0.00 7.74 43.82 9.51 96.70 79 0.177 5 0.495 0.673 0.00 7.74 58.11 9.51 96.71 79 0.177 5 0.495 0.673
- visualVM
When CPU high, can use visualVM find out the most long run method.
When CPU low, can get some stack sample, see where is the block point.
queue.put? queue.take?
- nmon great tool to monitor CPU, memory, network, disks...
** OTHERS *** JAVA AS SCRIPT aim: write business calculation/logic to script problem: java call Groovy/Python was so slow. - write business calculation/logic to file XXX.java, then compile and load use customized class loader. - can reload java class, after business calculation/logic changed, without restart Java instance. - use script to trace data
*** NETWORK
- ||CPU balance for network interfaces.
For some machine with multi CPU, if not properly configured,
all network interfaces' interrupts will goto CPU0 (like following show).
Then network performance will be restricted by power of CPU0. see here for the solution.
$ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
65: 20041 0 0 0 IR-PCI-MSI-edge eth0-tx-0
66: 20232 0 0 0 IR-PCI-MSI-edge eth0-tx-1
67: 20105 0 0 0 IR-PCI-MSI-edge eth0-tx-2
68: 20423 0 0 0 IR-PCI-MSI-edge eth0-tx-3
69: 21036 0 0 0 IR-PCI-MSI-edge eth0-rx-0
70: 20201 0 0 0 IR-PCI-MSI-edge eth0-rx-1
71: 20587 0 0 0 IR-PCI-MSI-edge eth0-rx-2
72: 20853 0 0 0 IR-PCI-MSI-edge eth0-rx-3
- set_thread_affinity ||If your system's performance is not stable, you can try set_thread_affinity. Restricting a process to run on a single CPU also avoids the performance cost caused by the cache invalidation that occurs when a process ceases to execute on one CPU and then recommences execution on a different CPU.int sched_setaffinity(pid_t pid,size_t cpusetsize,cpu_set_t *mask);- send several packages in a Big package. - pfring - Google Protocol Buffer
*** thread local storage to prevent new objects
JAVA: ThreadLocal
gcc: __thread int i;
- use array if possible
- use hash dispatch to avoid locking
- log sampling, if log is not important, then use queue.offer()
- keep thread count low.
- use message loop instead of timers
- less exception
- double buffer queue
** THE END
_______________________________________________________________________
_______________________________________________________________________
Subscribe to:
Posts (Atom)