Aug
8
初来日本的第一个冬天，见到中小学生们的短裤短裙，心里是免不了有点惊奇的。虽说自己高中时代也不惜落下风湿的毛病，在东北的寒风中扮酷穿单裤，但眼见那些年纪更小的童男童女光着两腿，不禁心生同情。问其缘由，有人说是为了从小磨其筋骨，不能不令人肃然起敬，并慨叹吾国之小皇帝们过于优乐。不过，若把目光投向稍大些的女中学生们，似乎又不仅是这个高尚的原因，因为那短裙的边际实在有点太高，有时简直到了大腿与臀部的交界。这不能说是一个为了砺炼意志的必要手段。
我们这一代人对日本女中学生制服的印象，大约始自电视剧《血疑》中山口百惠扮演的幸子。来到日本之后，才了解了它现在所蕴含的丰富意义，以及寄托了多少男性的粉色遐想。
在新宿、涉谷等繁华地带，时常能看到三两身穿校服的女中学生无所事事地伫立，而身边逛荡着个别上班族打扮的中年男子。有时，那男子会趋前搭讪，女孩们偶尔会发出肆无忌惮的大笑。女孩们的一个显著特点是超短的裙子，不管是穿着中文称之为“泡泡袜”的白色长袜（一般是将它叠在小腿部分，仿佛绑腿），还是黑色或蓝色半长袜，总之两截大腿是一定要露出来的，宛如刻意招展的旗帜。
起初，我对这些叽叽喳喳的女中学生和色眼迷离的男人们绝无好感，把她们看作社会公害。但随着年纪增长，我渐渐学会用一点理解的态度去看待。川端康成在《睡美人》中用细腻的笔触描述了一位老男人对年轻女性肉体的畸恋，在那些中年男人脸上，我似乎看到了同样的对衰老和死亡的由衷恐慌。而反过来打量女中学生们，我看到的是一种不知所措，混杂着对青春仿佛陡然暴富之后的挥霍欲和困惑感。
事实上，九十年代后期以来，日本女中学生的裙子的底边开始从及膝不断向上提高，算下来差不多每年一厘米。不知道这与泡沫经济的破灭有无联系，但肯定昭示了世风的潜在变化。许多外国游客都注意到了日本女中学生们展露出的大腿，而日本舆论也表达了对其所暗示的“援助交际”现象的忧虑。于是，如何让女中学生们把不断缩短的裙子长度放下来，俨然成了一个重要的社会问题。
学校是超短裙之争的第一线战场。不过，另一个有趣的数据是日本中小学教师猥亵学生案件的增长，2004年达到历史纪录的190余人，去年公布的资料则是下降到通常水准的160余人。这或许充分体现出了此一斗争的严酷性与复杂性。不久前的一个话题是新潟县的高中制作了劝说女生们把裙摆放长的海报，可见校方的用心良苦，然而据说女生们反应冷淡，认为“此举毫无意义”。
女中学生的大腿，俨然成了少女们的亚文化象征。但来日的中国人看来，自有不理解之处。我曾多次听到陪同的访客表示诧异，有男有女，他们不是诧异于日本女生的裙子之短，而是那双腿的“难看”：“她怎么敢穿这样的超短裙？”确实，大腿现象若有有一点值得称道的，便是众腿平等。姑娘们不管自己的腿多“难看”，也照样坦率露之。有的女生大腿粗如水桶，小腿孔武有力，据说在中国若敢穿超短裙恐怕要被人公然耻笑，在日本却可以安之若素。
进一步说来，日本女生公开袒露大腿可以说是为了迎合男性的意淫，将女性置于被亵玩的角色；可中国人对女性“腿型”（外貌）的苛求，实在是异曲同工甚至犹有过之。况且，穿着超短裙也并不一定就意味着性的暗示，一位新潟女高中生说：“我把裙子卷短并不是为了显得可爱，而是因为大家都这么做。”
大家都这么做。这才是问题的关键所在，即日本人强烈的群体性意识。接下来，我们就说说日本人的“跟风”嗜好。
我们这一代人对日本女中学生制服的印象，大约始自电视剧《血疑》中山口百惠扮演的幸子。来到日本之后，才了解了它现在所蕴含的丰富意义，以及寄托了多少男性的粉色遐想。
在新宿、涉谷等繁华地带，时常能看到三两身穿校服的女中学生无所事事地伫立，而身边逛荡着个别上班族打扮的中年男子。有时，那男子会趋前搭讪，女孩们偶尔会发出肆无忌惮的大笑。女孩们的一个显著特点是超短的裙子，不管是穿着中文称之为“泡泡袜”的白色长袜（一般是将它叠在小腿部分，仿佛绑腿），还是黑色或蓝色半长袜，总之两截大腿是一定要露出来的，宛如刻意招展的旗帜。
起初，我对这些叽叽喳喳的女中学生和色眼迷离的男人们绝无好感，把她们看作社会公害。但随着年纪增长，我渐渐学会用一点理解的态度去看待。川端康成在《睡美人》中用细腻的笔触描述了一位老男人对年轻女性肉体的畸恋，在那些中年男人脸上，我似乎看到了同样的对衰老和死亡的由衷恐慌。而反过来打量女中学生们，我看到的是一种不知所措，混杂着对青春仿佛陡然暴富之后的挥霍欲和困惑感。
事实上，九十年代后期以来，日本女中学生的裙子的底边开始从及膝不断向上提高，算下来差不多每年一厘米。不知道这与泡沫经济的破灭有无联系，但肯定昭示了世风的潜在变化。许多外国游客都注意到了日本女中学生们展露出的大腿，而日本舆论也表达了对其所暗示的“援助交际”现象的忧虑。于是，如何让女中学生们把不断缩短的裙子长度放下来，俨然成了一个重要的社会问题。
学校是超短裙之争的第一线战场。不过，另一个有趣的数据是日本中小学教师猥亵学生案件的增长，2004年达到历史纪录的190余人，去年公布的资料则是下降到通常水准的160余人。这或许充分体现出了此一斗争的严酷性与复杂性。不久前的一个话题是新潟县的高中制作了劝说女生们把裙摆放长的海报，可见校方的用心良苦，然而据说女生们反应冷淡，认为“此举毫无意义”。
女中学生的大腿，俨然成了少女们的亚文化象征。但来日的中国人看来，自有不理解之处。我曾多次听到陪同的访客表示诧异，有男有女，他们不是诧异于日本女生的裙子之短，而是那双腿的“难看”：“她怎么敢穿这样的超短裙？”确实，大腿现象若有有一点值得称道的，便是众腿平等。姑娘们不管自己的腿多“难看”，也照样坦率露之。有的女生大腿粗如水桶，小腿孔武有力，据说在中国若敢穿超短裙恐怕要被人公然耻笑，在日本却可以安之若素。
进一步说来，日本女生公开袒露大腿可以说是为了迎合男性的意淫，将女性置于被亵玩的角色；可中国人对女性“腿型”（外貌）的苛求，实在是异曲同工甚至犹有过之。况且，穿着超短裙也并不一定就意味着性的暗示，一位新潟女高中生说：“我把裙子卷短并不是为了显得可爱，而是因为大家都这么做。”
大家都这么做。这才是问题的关键所在，即日本人强烈的群体性意识。接下来，我们就说说日本人的“跟风”嗜好。
Jul
30
1. Understanding Internal 3proxy structure
3proxy is implemented as multithread application. Server model is implemented as “one connection – one thread”. It means, for every client connection new thread is created. This model is effective enough under Windows, because it allows it avoid thread creation on asynchronous operations, yet under most POSIX systems this model can not be considered as most efficient. It’s planned for (very far in future) release to implement more efficient model, where single thread can serve few clients.
1.1 main thread:
3proxy begins with main thread. This thread parses configuration file and starts main loop. During configuration file parsing struct extaparam conf; structure is filled and service threads are started.
Main loop cycle takes approximately 1 second and does these tasks:
- re-reads configuration file, if necessary
- performs scheduled tasks
- monitors files (‘monitor’ command), approx. once in a minute
- rotates main logfile
- dumps counters to file, approx. once in a minute
- performs termination, if required
It’s guaranteed every configuration and schedule command is executed from the same thread.
Main thread is implemented in 3proxy.c
1.2 service thread
Service threads are started immediately, than service command (e.g. ‘proxy’ or ‘socks’) are found during configuration file parsing. Each command creates new thread. Thread does these tasks:
- parses service command arguments and fills ‘struct srvparam srv’ structure with service configuration and ‘struct clientparam defparam’ structure with default client configuration
- initializes filters (filter_open)
- creates and initializes listening service socket
- enters into service loop
- terminates filters (filter_close)
service loop:
- checks for configuration reload (approximately every second), thread exits if configuration reloaded or 3proxy is in terminating state.
- accepts client connection and creates ‘struct clientparam newparam’ structure with client configuration
- creates/checks client filters (filter_client)
- creates client thread with newly created ‘struct clientparam newparam’
service threads are implemented in proxymain.c
Please note: struct clientparam is freed and filter_clear is executed from different (client) thread.
1.3 client thread
Client threads are started from service thread. Client thread:
- reads client request (except portmappers) with authentication information and request headers (if any).
- filters request (if any) with filter_request
- filters headers (if any) with filter_header_cli
- performs authentication and authorization
- established connection with server
- sends request to server
- filters server headers (if any)
- maps client end server sockets to transmit data between client and server
- logs request. Global counters are also updated on this operation
- clears client filters (filter_clear)
- frees struct clientparam data
in some point client thread may loop to process few client requests from the same connection (e.g. HTTP ‘established’ connection in ‘proxy’).
Socket mapping does:
- caches data in internal client and server buffers
- delays data transmit to limit bandwidth
- performs data filtering (filter_data_cli / filter_data_srv)
client threads are implemented in proxy.c, socks.c, pop3p.c etc.
2. Hacking into 3proxy code with plugins
2.1 What is 3proxy plugin
3proxy plugin is any dynamic/shared library. There is no specific requirement for plugin, actually you can load any dynamic library with ‘plugin’ command. No linking with any libraries are required. However, to interoperate with 3proxy dynamic library must have an export function 3proxy may call to pass the structure with required information.
typedef int (*PLUGINFUNC) (struct pluginlink *pluginlink, int argc, char** argv);
struct pluginlink is a structure with export information, explained later, argc and argv are argument counter and array of arguments of “plugin” command. Plugin should report it’s status with integer return value. 0 is success, positive value indicates non-recoverable error, 3proxy do not parse rest of configuration and enters into termination state, negative value indicates recoverable value, 3proxy logs warning (if possible). In case of C++, all 3proxy functions/structures must be extern “C”.
All 3proxy structures/functions descriptions are located in structures.h
2.2 Understanding pluginlink structure
Because there is no linking between 3proxy and plugin, all 3proxy functions and structures are passed with pluginlink structure. Pluginlink is actually a collection of pointers to 3proxy internal structures and functions. Because pluginlink is constantly extending, you should see it’s definitions in structures.h.
most important are:
struct symbol symbols;
“symbols” is a kind of name/value export table, made as a list. It can be used by plugins to exchange information and functions between plugins, e.g. to export functions from one plugin to another, where pluginlink is useless, because it’s static. It’s quite simple:
struct symbol {
struct symbol *next;
char * name;
void * value;
};
name – is a name of function or structure
value – is it’s value.
use pluginlink->fundbyname function to lookup, e.g.
anotherplugindata = pluginlink->findbyname(“anotherplugindata”);
To export something from your plugin, add your structure to this list.
struct extparam *conf;
pointer to conf structure, it holds all current 3proxy configuration
2.3 How to get control within plugin
There are few points you can get control for your plugin, after it’s loaded with ‘plugin’ command.
2.3.1 Adding configuration command processor with struct command structure
A list of configuration file command, available from 3proxy.cfg is extendable. Each command is defined by struct commands:
struct commands {
struct commands *next;
char * command;
int (* handler)(int argc, unsigned char ** argv);
int minargs;
int maxargs;
};
struct commands *next - next element in list
char * command – command name
int (* handler)(int argc, unsigned char ** argv) – command handler. It’s called than ‘command’ is found in configuration files, argc is a number of arguments, counting command itself, argv is array of arguments.
minargs – minimum number of arguments command support (>= 1)
maxargs – maximum number of arguments command support, 0 means infinity.
Handler return value of 0 indicates command is successfully processed. Positive return value indicates non-recoverable error, 3proxy enters termination state. Negative value indicates 3proxy to continue to process command list, it makes it possible to set few handlers for the same command.
A list of the command is pointed by pluginlinks->commandhandlers; you must insert you command after first one (do not replace pluginlink->commandhandlers). It’s guaranteed at least 1 dummy command is always present.
Example:
int mycommandhandler(int argc, unsigned char **argv);
struct commands mycommand;
mycommand.command = “mycommand”;
mycommand.handler = mycommandhandler;
mycommand.intargs = 1;
mycommand.intargs = 2;
mycommand.next = pluginslinks->commandhandlers->next;
pluginslink->commandhandlers->next = &mycommand;
Adds processor for “mycommand” command with zero on one arguments.
Adding configuration command is useful, if your plugin expects configuration data.
2.3.2 Adding authentication method with struct auth
3proxy supports authentication and authorization. Authentication process determines user account (for example by username and password), authorization checks, if user account has a right to access given resource and optionally establishes a connection, if required.
‘auth’ command combines both authentication and authorization method. It’s extandable with struct auth list:
struct auth {
struct auth *next;
AUTHFUNC authenticate;
AUTHFUNC authorize;
char * desc;
};
char * desc – name of authentication/authorization method
authenticate – name of authentication function
authorize – name of authorization function
pluginlink->authfuncs points to list of authenticataction structures. Like above, new structure must be inserted after fiest one (or to the end of the list).
First, authentication is called, if authentication indicates OK status (return value 0), authorization is called. Normally, ‘checkACL’ (pluginlink->checkACL) is called as authorization function to check user’s request matches to standard allow/deny rules. If for some reason you need to avoid this check, you should call pluginlink->alwaysauth to do some dirty job, like establishing outgoing connection.
typedef int (*AUTHFUNC)(struct clientparam * param);
is both authentication and authorization function. ‘struct clientparam’ holds all information about client connection, including username (param->username) and password (param->password).
Return value of 0 indicates successful authentication/authorization, 1 and 3 – authorization failed (access denied), use 3 in case you want to indicate access is explicitly denied and 3 in case there is no matching rule. 4,5,6,7,8 – authentication failed (e.g. username/password do not match). 4 indicates username does not present in request and must be requested, if possible. 5 indicates username found in request can not be found in user’s database/list, 6,7,8 – username does not match password for different authentication types. 10 – user exceeded some limits, e.g. traffic. You may use some different code to indicate internal problems.
Example:
int myauthfunc(struct clientparam *param);
struct auth myauth;
myauth.desc = “myauth”;
myauth.authenticate = myauthfunc;
myauth.authorize = pluginlink->checkACL;
myauth->next = pluginlink->authfuncs->next;
pluginlink->authfuncs->next = &myauth;
Installs “myauthfunc” as authentication function. There is no need to add ‘auth’ command processor for new authentication type, it’s processed by standard ‘auth’ command processor.
2.3.3 Adding scheduled functions
Scheduled functions are described by this structure:
typedef enum {NONE, MINUTELY, HOURLY, DAILY, WEEKLY, MONTHLY, ANNUALLY, NEVER} ROTATION;
struct schedule {
struct schedule *next;
ROTATION type;
void *data;
int (*function)(void *);
time_t start_time;
};
int (*function)(void *) – scheduled function
void *data – this pointer will be passed as an argument to scheduled functions
ROTATION type – defines how often function is called (once in a minute, hour, etc).
start_time – time to begin using of scheduled function
Scheduled functions are called every ‘type’ interval after start_time and also on reloading configuration and going to termination state.
Schedule function return value of 1 means function must be removed from the schedule. 3proxy doesn’t free struct schedule.
Schedule list can be empty. Pointer to schedule is pointed by struct schedule ** schedule; in pluginlink.
Example:
int myschedfunc(void * data);
struct schedule myschedule;
myschedule.data = “somethinghere”;
myschedule.function = myschedfunc;
myschedule.type = MINUTELY;
myschedule.starttime = 0;
myschedule.next = *pluginlink->schedule;
*pluginlink->schedule = myschedule;
NOTE: time_t is different for different compilers. Make sure to compile plugin and 3proxy with same compiler.
2.3.4 Filters API
3proxy has filters API, you can use, to process client request and data flowing through proxy. It should be noted, that currently 3proxy doesn’t provide filters with any useful data conversion, so, it’s filter’s task to find data in data flow. In case filter modifies some data, it’s filter’s task again to assure that everything’s fine. If you know some filtering API like MILTER, you will find 3proxy filters very same.
typedef enum {
PASS,
CONTINUE,
HANDLED,
REJECT,
REMOVE
} FILTER_ACTION;
typedef void* FILTER_OPEN(void * idata, struct srvparam * param);
typedef FILTER_ACTION FILTER_CLIENT(void *fo, struct clientparam * param, void** fc);
typedef FILTER_ACTION FILTER_BUFFER(void *fc, struct clientparam * param, unsigned char ** buf_p, int * bufsize_p, int offset, int * length_p);
typedef void FILTER_CLOSE(void *fo);
struct filter {
struct filter * next;
char * instance;
void * data;
FILTER_OPEN *filter_open;
FILTER_CLIENT *filter_client;
FILTER_BUFFER *filter_request;
FILTER_BUFFER *filter_header_cli;
FILTER_BUFFER *filter_header_srv;
FILTER_BUFFER *filter_data_cli;
FILTER_BUFFER *filter_data_srv;
FILTER_CLOSE *filter_clear;
FILTER_CLOSE *filter_close;
};
char * instance – is some instance identifier. You can use it to find required filter in the list. 3proxy itself doesn’t use this field.
void * data – this parameter is passed to filter_open function.
The rest are filtering functions. Section 1 explains where and then each filter is called. data should not be NULL.
filter_open must always be defined, if you want filter to be ever used. It’s called then new service is created and is given “data” from struct filter and struct srvparam (parsed service configuration). If filter_open for some filter returns NULL, filter will not be used for this service. Non-NULL return value will be used as “fo” parameter for every call to filter_client.
filter_client is called upon client connect (before any data is sent/received). It’s good place to filter client by IP (and is not good place to filter it by hostname, because this operation takes a long time, 3proxy will not be able to accept new connection). fo is a data pointer received from filter_open, param – newly created clientparam structure, fc is return parameter filter_open must initialize, it will be used as an argument to FILTER_BUFFER functions. PASS return value means this filter will be used for this client request. CONTINUE says to install filters. On different values client connection is closed and no client thread is created.
filter_request, filter_header_cli, filter_header_srv, filter_data_cli, filter_data_srv are used to process request and data received from client and server.
char ** buf_p is a pointer to current buffer, int * bufsize_p is a pointer to it’s size. In case you change some data and it doesn’t fit to current buffer, you may allocate new buffer (with pluginlink->myalloc), copy data from old buffer, free old buffer (with pluginlink->myfree) and set new values for *buf_p and *bufsize_p.
int * offset_p offset of the new data in the buffer, int * length_p length of all data in the buffer. You should filter only (*length_p - *offset_p) characters starting from (*buf_p + *offset_p).
filter_clear is called for each successfule filter_client and should be used to free allocated resources
filter_close is called for each successful filter_open for the same reason
An example of filter API usage you can find in PCREPlugin (see plugins/ PCREPlugin/ pcre_plugin.c).
Note: if param->nooverwritefilter is set for FILTER_BUFFER functions, filter may change data in the buffer, but must not change data length. This flag may be set, if data size if already known and is sent to the client.
2.3.5 Replacing log functions, traffic counting functions, bandwidth limitation functions
Log, traffic and bandwidth function can be directly replaced in any over place. All functions may be replaced in conf (pluginlink->conf->logfunc, pluginlink->conf->bandlimfunc, pluginlink->conf->trafcountfunc). In this case, these functions will be used for services started after the changes are made. logfunc may also be changed for struct srvparam (e.g. within filter_open), bandlimfunc and trafcountfunc may be changed in struct clientparam for every client individually (e.g. within filter_client).
typedef void (*LOGFUNC)(struct clientparam * param, const unsigned char * test);
typedef void (*TRAFCOUNTFUNC)(struct clientparam * param);
typedef unsigned (*BANDLIMFUNC)(struct clientparam * param, unsigned nbytesin, unsigned nbytesout);
struct clientparam * param – information about client request
char * text - text string (e.g. request)
nbytesin, nbytesout – number of bytes received from / send to server. bandlimfunc returns delay in milliseconds.
3proxy is implemented as multithread application. Server model is implemented as “one connection – one thread”. It means, for every client connection new thread is created. This model is effective enough under Windows, because it allows it avoid thread creation on asynchronous operations, yet under most POSIX systems this model can not be considered as most efficient. It’s planned for (very far in future) release to implement more efficient model, where single thread can serve few clients.
1.1 main thread:
3proxy begins with main thread. This thread parses configuration file and starts main loop. During configuration file parsing struct extaparam conf; structure is filled and service threads are started.
Main loop cycle takes approximately 1 second and does these tasks:
- re-reads configuration file, if necessary
- performs scheduled tasks
- monitors files (‘monitor’ command), approx. once in a minute
- rotates main logfile
- dumps counters to file, approx. once in a minute
- performs termination, if required
It’s guaranteed every configuration and schedule command is executed from the same thread.
Main thread is implemented in 3proxy.c
1.2 service thread
Service threads are started immediately, than service command (e.g. ‘proxy’ or ‘socks’) are found during configuration file parsing. Each command creates new thread. Thread does these tasks:
- parses service command arguments and fills ‘struct srvparam srv’ structure with service configuration and ‘struct clientparam defparam’ structure with default client configuration
- initializes filters (filter_open)
- creates and initializes listening service socket
- enters into service loop
- terminates filters (filter_close)
service loop:
- checks for configuration reload (approximately every second), thread exits if configuration reloaded or 3proxy is in terminating state.
- accepts client connection and creates ‘struct clientparam newparam’ structure with client configuration
- creates/checks client filters (filter_client)
- creates client thread with newly created ‘struct clientparam newparam’
service threads are implemented in proxymain.c
Please note: struct clientparam is freed and filter_clear is executed from different (client) thread.
1.3 client thread
Client threads are started from service thread. Client thread:
- reads client request (except portmappers) with authentication information and request headers (if any).
- filters request (if any) with filter_request
- filters headers (if any) with filter_header_cli
- performs authentication and authorization
- established connection with server
- sends request to server
- filters server headers (if any)
- maps client end server sockets to transmit data between client and server
- logs request. Global counters are also updated on this operation
- clears client filters (filter_clear)
- frees struct clientparam data
in some point client thread may loop to process few client requests from the same connection (e.g. HTTP ‘established’ connection in ‘proxy’).
Socket mapping does:
- caches data in internal client and server buffers
- delays data transmit to limit bandwidth
- performs data filtering (filter_data_cli / filter_data_srv)
client threads are implemented in proxy.c, socks.c, pop3p.c etc.
2. Hacking into 3proxy code with plugins
2.1 What is 3proxy plugin
3proxy plugin is any dynamic/shared library. There is no specific requirement for plugin, actually you can load any dynamic library with ‘plugin’ command. No linking with any libraries are required. However, to interoperate with 3proxy dynamic library must have an export function 3proxy may call to pass the structure with required information.
typedef int (*PLUGINFUNC) (struct pluginlink *pluginlink, int argc, char** argv);
struct pluginlink is a structure with export information, explained later, argc and argv are argument counter and array of arguments of “plugin” command. Plugin should report it’s status with integer return value. 0 is success, positive value indicates non-recoverable error, 3proxy do not parse rest of configuration and enters into termination state, negative value indicates recoverable value, 3proxy logs warning (if possible). In case of C++, all 3proxy functions/structures must be extern “C”.
All 3proxy structures/functions descriptions are located in structures.h
2.2 Understanding pluginlink structure
Because there is no linking between 3proxy and plugin, all 3proxy functions and structures are passed with pluginlink structure. Pluginlink is actually a collection of pointers to 3proxy internal structures and functions. Because pluginlink is constantly extending, you should see it’s definitions in structures.h.
most important are:
struct symbol symbols;
“symbols” is a kind of name/value export table, made as a list. It can be used by plugins to exchange information and functions between plugins, e.g. to export functions from one plugin to another, where pluginlink is useless, because it’s static. It’s quite simple:
struct symbol {
struct symbol *next;
char * name;
void * value;
};
name – is a name of function or structure
value – is it’s value.
use pluginlink->fundbyname function to lookup, e.g.
anotherplugindata = pluginlink->findbyname(“anotherplugindata”);
To export something from your plugin, add your structure to this list.
struct extparam *conf;
pointer to conf structure, it holds all current 3proxy configuration
2.3 How to get control within plugin
There are few points you can get control for your plugin, after it’s loaded with ‘plugin’ command.
2.3.1 Adding configuration command processor with struct command structure
A list of configuration file command, available from 3proxy.cfg is extendable. Each command is defined by struct commands:
struct commands {
struct commands *next;
char * command;
int (* handler)(int argc, unsigned char ** argv);
int minargs;
int maxargs;
};
struct commands *next - next element in list
char * command – command name
int (* handler)(int argc, unsigned char ** argv) – command handler. It’s called than ‘command’ is found in configuration files, argc is a number of arguments, counting command itself, argv is array of arguments.
minargs – minimum number of arguments command support (>= 1)
maxargs – maximum number of arguments command support, 0 means infinity.
Handler return value of 0 indicates command is successfully processed. Positive return value indicates non-recoverable error, 3proxy enters termination state. Negative value indicates 3proxy to continue to process command list, it makes it possible to set few handlers for the same command.
A list of the command is pointed by pluginlinks->commandhandlers; you must insert you command after first one (do not replace pluginlink->commandhandlers). It’s guaranteed at least 1 dummy command is always present.
Example:
int mycommandhandler(int argc, unsigned char **argv);
struct commands mycommand;
mycommand.command = “mycommand”;
mycommand.handler = mycommandhandler;
mycommand.intargs = 1;
mycommand.intargs = 2;
mycommand.next = pluginslinks->commandhandlers->next;
pluginslink->commandhandlers->next = &mycommand;
Adds processor for “mycommand” command with zero on one arguments.
Adding configuration command is useful, if your plugin expects configuration data.
2.3.2 Adding authentication method with struct auth
3proxy supports authentication and authorization. Authentication process determines user account (for example by username and password), authorization checks, if user account has a right to access given resource and optionally establishes a connection, if required.
‘auth’ command combines both authentication and authorization method. It’s extandable with struct auth list:
struct auth {
struct auth *next;
AUTHFUNC authenticate;
AUTHFUNC authorize;
char * desc;
};
char * desc – name of authentication/authorization method
authenticate – name of authentication function
authorize – name of authorization function
pluginlink->authfuncs points to list of authenticataction structures. Like above, new structure must be inserted after fiest one (or to the end of the list).
First, authentication is called, if authentication indicates OK status (return value 0), authorization is called. Normally, ‘checkACL’ (pluginlink->checkACL) is called as authorization function to check user’s request matches to standard allow/deny rules. If for some reason you need to avoid this check, you should call pluginlink->alwaysauth to do some dirty job, like establishing outgoing connection.
typedef int (*AUTHFUNC)(struct clientparam * param);
is both authentication and authorization function. ‘struct clientparam’ holds all information about client connection, including username (param->username) and password (param->password).
Return value of 0 indicates successful authentication/authorization, 1 and 3 – authorization failed (access denied), use 3 in case you want to indicate access is explicitly denied and 3 in case there is no matching rule. 4,5,6,7,8 – authentication failed (e.g. username/password do not match). 4 indicates username does not present in request and must be requested, if possible. 5 indicates username found in request can not be found in user’s database/list, 6,7,8 – username does not match password for different authentication types. 10 – user exceeded some limits, e.g. traffic. You may use some different code to indicate internal problems.
Example:
int myauthfunc(struct clientparam *param);
struct auth myauth;
myauth.desc = “myauth”;
myauth.authenticate = myauthfunc;
myauth.authorize = pluginlink->checkACL;
myauth->next = pluginlink->authfuncs->next;
pluginlink->authfuncs->next = &myauth;
Installs “myauthfunc” as authentication function. There is no need to add ‘auth’ command processor for new authentication type, it’s processed by standard ‘auth’ command processor.
2.3.3 Adding scheduled functions
Scheduled functions are described by this structure:
typedef enum {NONE, MINUTELY, HOURLY, DAILY, WEEKLY, MONTHLY, ANNUALLY, NEVER} ROTATION;
struct schedule {
struct schedule *next;
ROTATION type;
void *data;
int (*function)(void *);
time_t start_time;
};
int (*function)(void *) – scheduled function
void *data – this pointer will be passed as an argument to scheduled functions
ROTATION type – defines how often function is called (once in a minute, hour, etc).
start_time – time to begin using of scheduled function
Scheduled functions are called every ‘type’ interval after start_time and also on reloading configuration and going to termination state.
Schedule function return value of 1 means function must be removed from the schedule. 3proxy doesn’t free struct schedule.
Schedule list can be empty. Pointer to schedule is pointed by struct schedule ** schedule; in pluginlink.
Example:
int myschedfunc(void * data);
struct schedule myschedule;
myschedule.data = “somethinghere”;
myschedule.function = myschedfunc;
myschedule.type = MINUTELY;
myschedule.starttime = 0;
myschedule.next = *pluginlink->schedule;
*pluginlink->schedule = myschedule;
NOTE: time_t is different for different compilers. Make sure to compile plugin and 3proxy with same compiler.
2.3.4 Filters API
3proxy has filters API, you can use, to process client request and data flowing through proxy. It should be noted, that currently 3proxy doesn’t provide filters with any useful data conversion, so, it’s filter’s task to find data in data flow. In case filter modifies some data, it’s filter’s task again to assure that everything’s fine. If you know some filtering API like MILTER, you will find 3proxy filters very same.
typedef enum {
PASS,
CONTINUE,
HANDLED,
REJECT,
REMOVE
} FILTER_ACTION;
typedef void* FILTER_OPEN(void * idata, struct srvparam * param);
typedef FILTER_ACTION FILTER_CLIENT(void *fo, struct clientparam * param, void** fc);
typedef FILTER_ACTION FILTER_BUFFER(void *fc, struct clientparam * param, unsigned char ** buf_p, int * bufsize_p, int offset, int * length_p);
typedef void FILTER_CLOSE(void *fo);
struct filter {
struct filter * next;
char * instance;
void * data;
FILTER_OPEN *filter_open;
FILTER_CLIENT *filter_client;
FILTER_BUFFER *filter_request;
FILTER_BUFFER *filter_header_cli;
FILTER_BUFFER *filter_header_srv;
FILTER_BUFFER *filter_data_cli;
FILTER_BUFFER *filter_data_srv;
FILTER_CLOSE *filter_clear;
FILTER_CLOSE *filter_close;
};
char * instance – is some instance identifier. You can use it to find required filter in the list. 3proxy itself doesn’t use this field.
void * data – this parameter is passed to filter_open function.
The rest are filtering functions. Section 1 explains where and then each filter is called. data should not be NULL.
filter_open must always be defined, if you want filter to be ever used. It’s called then new service is created and is given “data” from struct filter and struct srvparam (parsed service configuration). If filter_open for some filter returns NULL, filter will not be used for this service. Non-NULL return value will be used as “fo” parameter for every call to filter_client.
filter_client is called upon client connect (before any data is sent/received). It’s good place to filter client by IP (and is not good place to filter it by hostname, because this operation takes a long time, 3proxy will not be able to accept new connection). fo is a data pointer received from filter_open, param – newly created clientparam structure, fc is return parameter filter_open must initialize, it will be used as an argument to FILTER_BUFFER functions. PASS return value means this filter will be used for this client request. CONTINUE says to install filters. On different values client connection is closed and no client thread is created.
filter_request, filter_header_cli, filter_header_srv, filter_data_cli, filter_data_srv are used to process request and data received from client and server.
char ** buf_p is a pointer to current buffer, int * bufsize_p is a pointer to it’s size. In case you change some data and it doesn’t fit to current buffer, you may allocate new buffer (with pluginlink->myalloc), copy data from old buffer, free old buffer (with pluginlink->myfree) and set new values for *buf_p and *bufsize_p.
int * offset_p offset of the new data in the buffer, int * length_p length of all data in the buffer. You should filter only (*length_p - *offset_p) characters starting from (*buf_p + *offset_p).
filter_clear is called for each successfule filter_client and should be used to free allocated resources
filter_close is called for each successful filter_open for the same reason
An example of filter API usage you can find in PCREPlugin (see plugins/ PCREPlugin/ pcre_plugin.c).
Note: if param->nooverwritefilter is set for FILTER_BUFFER functions, filter may change data in the buffer, but must not change data length. This flag may be set, if data size if already known and is sent to the client.
2.3.5 Replacing log functions, traffic counting functions, bandwidth limitation functions
Log, traffic and bandwidth function can be directly replaced in any over place. All functions may be replaced in conf (pluginlink->conf->logfunc, pluginlink->conf->bandlimfunc, pluginlink->conf->trafcountfunc). In this case, these functions will be used for services started after the changes are made. logfunc may also be changed for struct srvparam (e.g. within filter_open), bandlimfunc and trafcountfunc may be changed in struct clientparam for every client individually (e.g. within filter_client).
typedef void (*LOGFUNC)(struct clientparam * param, const unsigned char * test);
typedef void (*TRAFCOUNTFUNC)(struct clientparam * param);
typedef unsigned (*BANDLIMFUNC)(struct clientparam * param, unsigned nbytesin, unsigned nbytesout);
struct clientparam * param – information about client request
char * text - text string (e.g. request)
nbytesin, nbytesout – number of bytes received from / send to server. bandlimfunc returns delay in milliseconds.
Jul
15
本人现在开发的所有项目都使用postgresql，应用下来对它很是满意，现就影响postgresql性能的几个重要参数介绍如下，希望对PG的初学者有所帮助，如果你在实际应用中遇到什么问题，可给我留言，我们一起解决：
PG的配置文件是数据库目录下的postgresql.conf文件，8.0以后的版本可支持K，M，G这样的参数，只要修改相应参数后重新启动PG服务就OK了。
shared_buffers：这是最重要的参数，postgresql通过shared_buffers和内核和磁盘打交道，因此应该尽量大，让更多的数据缓存在shared_buffers中。通常设置为实际RAM的10％是合理的，比如50000(400M)
work_mem： 在pgsql 8.0之前叫做sort_mem。postgresql在执行排序操作时，会根据work_mem的大小决定是否将一个大的结果集拆分为几个小的和 work_mem查不多大小的临时文件。显然拆分的结果是降低了排序的速度。因此增加work_mem有助于提高排序的速度。通常设置为实际RAM的2% -4%，根据需要排序结果集的大小而定，比如81920(80M)
effective_cache_size：是postgresql能够使用的最大缓存，这个数字对于独立的pgsql服务器而言应该足够大，比如4G的内存，可以设置为3.5G(437500)
maintence_work_mem：这里定义的内存只是在CREATE INDEX, VACUUM等时用到，因此用到的频率不高，但是往往这些指令消耗比较多的资源，因此应该尽快让这些指令快速执行完毕：给 maintence_work_mem大的内存，比如512M(524288)
max_connections： 通常，max_connections的目的是防止max_connections * work_mem超出了实际内存大小。比如，如果将work_mem设置为实际内存的2%大小，则在极端情况下，如果有50个查询都有排序要求，而且都使用2％的内存，则会导致swap的产生，系统性能就会大大降低。当然，如果有4G的内存，同时出现50个如此大的查询的几率应该是很小的。不过，要清楚 max_connections和work_mem的关系。
PG的配置文件是数据库目录下的postgresql.conf文件，8.0以后的版本可支持K，M，G这样的参数，只要修改相应参数后重新启动PG服务就OK了。
shared_buffers：这是最重要的参数，postgresql通过shared_buffers和内核和磁盘打交道，因此应该尽量大，让更多的数据缓存在shared_buffers中。通常设置为实际RAM的10％是合理的，比如50000(400M)
work_mem： 在pgsql 8.0之前叫做sort_mem。postgresql在执行排序操作时，会根据work_mem的大小决定是否将一个大的结果集拆分为几个小的和 work_mem查不多大小的临时文件。显然拆分的结果是降低了排序的速度。因此增加work_mem有助于提高排序的速度。通常设置为实际RAM的2% -4%，根据需要排序结果集的大小而定，比如81920(80M)
effective_cache_size：是postgresql能够使用的最大缓存，这个数字对于独立的pgsql服务器而言应该足够大，比如4G的内存，可以设置为3.5G(437500)
maintence_work_mem：这里定义的内存只是在CREATE INDEX, VACUUM等时用到，因此用到的频率不高，但是往往这些指令消耗比较多的资源，因此应该尽快让这些指令快速执行完毕：给 maintence_work_mem大的内存，比如512M(524288)
max_connections： 通常，max_connections的目的是防止max_connections * work_mem超出了实际内存大小。比如，如果将work_mem设置为实际内存的2%大小，则在极端情况下，如果有50个查询都有排序要求，而且都使用2％的内存，则会导致swap的产生，系统性能就会大大降低。当然，如果有4G的内存，同时出现50个如此大的查询的几率应该是很小的。不过，要清楚 max_connections和work_mem的关系。
Jul
15
行ったのは、以下のパラメータ
１．postgresql.conf
・shared_buffers （共有バッファ）
・sort_mem （ソートメモリ）
・wal_ buffers （トランザクションログバッファ）
・checkpoint_segments （チェックポイントセグメント）
・max_fsm_pages （FSM[Free Space Map]））
２．shmmax （カーネルの共有メモリ量）
/proc/sys/kernel/shmmax
/etc/sysctl.conf
今回は、キャッシュ機能を有効利用して、キャッシュヒット率を高めるのが大きな目的だったので、以下のサイトの手順によってパラメータに入れるべき数値を計算した。
１．postgresql.conf
・shared_buffers （共有バッファ）
・sort_mem （ソートメモリ）
・wal_ buffers （トランザクションログバッファ）
・checkpoint_segments （チェックポイントセグメント）
・max_fsm_pages （FSM[Free Space Map]））
２．shmmax （カーネルの共有メモリ量）
/proc/sys/kernel/shmmax
/etc/sysctl.conf
今回は、キャッシュ機能を有効利用して、キャッシュヒット率を高めるのが大きな目的だったので、以下のサイトの手順によってパラメータに入れるべき数値を計算した。
Jul
15
以下のサイトを参考にPostgresqlのインストールを行ったので、作業手順などのメモを備忘録として残しておく。
Postgresqlインストールの参考サイト
１．RPMですでにPostgresqlがインストールされていないかをチェック
# rpm -qa | grep postgres
インストールされていなかったので、削除せずに次へ。
２．postgresユーザーを作成
→セキュリティを高めるため、rootではなく、postgresでインストールと起動を行うためにユーザーを作成する。
# adduser postgres
# passwd postgres （このあと２回パスワードを入力する）
３．ソース展開用またはインストール用ディレクトリ作成
# mkdir /usr/local/src/postgresql-8.1.2
# chown postgres:postgres /usr/local/src/postgresql-8.1.2
# mkdir /usr/local/pgsql
# chown postgres:postgres /usr/local/pgsql
４．Postgresqlのダウンロード
Postgresql本家から必要なバージョンのファイルをダウンロード。
/usr/local/src に置いておく。
今回ダウンロードしたのは、バージョン8.1.2
５．Postgresが解凍できるようにパーミッションを変更
# cd /usr/local/src
# chmod 755 postgresql-8.1.2.tar.gz
６．rootからpostgresユーザーに切り変え、解凍する。
# su - postgres
$ cd /usr/local/src
$ tar xzvf postgresql-8.1.2.tar.gz
７．Makefile作成
$ cd postgresql-8.1.2
$ ./configure
ここで、「configure: error: readline library not found」というエラーが発生。
readlineライブラリがインストールされていないのが原因だった。
８．readlineライブラリのインストール
→これは、postgreユーザーではなく、rootでインストールを行う。
# cd /usr/local/src
# wget ftp://ftp.gnu.org/gnu/readline/readline-5.1.tar.gz
# tar zxvf readline-5.1.tar.gz
# cd readline-5.1
# ./configure
ここで、以下のようなエラーが発生。
checking test program... failed
configure: error:
*** Could not execute a simple test program. This may be a problem
*** related to locating shared libraries. Check the file 'config.log'
*** for the exact reason.
config.logをチェックしろということだったので、見てみると・・・
# less ./config.log
「./conftest: error while loading shared libraries: libreadline.so.5:cannot open shared object file: No such file or directory」
というエラーが発生していた。
google先生に聞くと、環境変数にLD_LIBRARY_PATHの設定をしたら解決できるとのことだったので、以下のように設定してから再度configureすると今度はエラーが出なかったので、そのままインストールを続ける。
# export LD_LIBRARY_PATH="$LD_LIBRARY_PATH":/usr/local/lib
# ./configure
# make
# make install
# ldconfig
参考サイト
参考サイト
９．再度、PostgresqlのMakefile作成
# su - postgres
$ cd postgresql-8.1.2
$ ./configure
$ make all
$ make check
$ make install
とすると、「 PostgreSQL installation complete.」と出たのでうまくいったみたい。
１０．環境設定
$ cd /home/postgres
$ vi .bashrc
として以下を追記する。
PATH="$PATH":/usr/local/pgsql/bin
export POSTGRES_HOME=/usr/local/pgsql
export PGLIB=$POSTGRES_HOME/lib
export PGDATA=$POSTGRES_HOME/data
export MANPATH="$MANPATH":$POSTGRES_HOME/man
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH":$PGLIB:/usr/local/lib
$ source ~/.bashrc で環境設定を反映させて、
$ printenv で設定が反映されているかを確認。
１１．データベースの初期化
$ initdb --encoding=EUC_JP --no-locale
１２．設定ファイルの修正
まずは、/usr/local/pgsql/data/postgresql.conf を以下のように修正
log_destination = 'syslog'
log_statement = 'all'
listen_addresses = '*'
次に、/usr/local/pgsql/data/pg_hba.conf を以下のように修正
host all all 192.168.0.0/24 trust
参考サイト
syslogの設定なども上記サイトのように修正。
１３．Postgresqlの起動
$ pg_ctl -w start
１４．Postgresqlの起動確認
$ ps ax | grep postgres
１５．Postgresqlの停止
$ pg_ctl stop
１６．サーバマシンが起動する時に自動的にPostgreSQLが起動する設定にする。
→これは、postgresユーザーではなく、rootで行う。
# cd /etc/rc.d/init.d/
# cp -p /usr/local/src/postgresql-8.1.2/contrib/start-scripts/linux postgresql
# chown root.root postgresql
# chmod 755 postgresql
# chkconfig --level 2345 postgresql on
Postgresqlインストールの参考サイト
１．RPMですでにPostgresqlがインストールされていないかをチェック
# rpm -qa | grep postgres
インストールされていなかったので、削除せずに次へ。
２．postgresユーザーを作成
→セキュリティを高めるため、rootではなく、postgresでインストールと起動を行うためにユーザーを作成する。
# adduser postgres
# passwd postgres （このあと２回パスワードを入力する）
３．ソース展開用またはインストール用ディレクトリ作成
# mkdir /usr/local/src/postgresql-8.1.2
# chown postgres:postgres /usr/local/src/postgresql-8.1.2
# mkdir /usr/local/pgsql
# chown postgres:postgres /usr/local/pgsql
４．Postgresqlのダウンロード
Postgresql本家から必要なバージョンのファイルをダウンロード。
/usr/local/src に置いておく。
今回ダウンロードしたのは、バージョン8.1.2
５．Postgresが解凍できるようにパーミッションを変更
# cd /usr/local/src
# chmod 755 postgresql-8.1.2.tar.gz
６．rootからpostgresユーザーに切り変え、解凍する。
# su - postgres
$ cd /usr/local/src
$ tar xzvf postgresql-8.1.2.tar.gz
７．Makefile作成
$ cd postgresql-8.1.2
$ ./configure
ここで、「configure: error: readline library not found」というエラーが発生。
readlineライブラリがインストールされていないのが原因だった。
８．readlineライブラリのインストール
→これは、postgreユーザーではなく、rootでインストールを行う。
# cd /usr/local/src
# wget ftp://ftp.gnu.org/gnu/readline/readline-5.1.tar.gz
# tar zxvf readline-5.1.tar.gz
# cd readline-5.1
# ./configure
ここで、以下のようなエラーが発生。
checking test program... failed
configure: error:
*** Could not execute a simple test program. This may be a problem
*** related to locating shared libraries. Check the file 'config.log'
*** for the exact reason.
config.logをチェックしろということだったので、見てみると・・・
# less ./config.log
「./conftest: error while loading shared libraries: libreadline.so.5:cannot open shared object file: No such file or directory」
というエラーが発生していた。
google先生に聞くと、環境変数にLD_LIBRARY_PATHの設定をしたら解決できるとのことだったので、以下のように設定してから再度configureすると今度はエラーが出なかったので、そのままインストールを続ける。
# export LD_LIBRARY_PATH="$LD_LIBRARY_PATH":/usr/local/lib
# ./configure
# make
# make install
# ldconfig
参考サイト
参考サイト
９．再度、PostgresqlのMakefile作成
# su - postgres
$ cd postgresql-8.1.2
$ ./configure
$ make all
$ make check
$ make install
とすると、「 PostgreSQL installation complete.」と出たのでうまくいったみたい。
１０．環境設定
$ cd /home/postgres
$ vi .bashrc
として以下を追記する。
PATH="$PATH":/usr/local/pgsql/bin
export POSTGRES_HOME=/usr/local/pgsql
export PGLIB=$POSTGRES_HOME/lib
export PGDATA=$POSTGRES_HOME/data
export MANPATH="$MANPATH":$POSTGRES_HOME/man
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH":$PGLIB:/usr/local/lib
$ source ~/.bashrc で環境設定を反映させて、
$ printenv で設定が反映されているかを確認。
１１．データベースの初期化
$ initdb --encoding=EUC_JP --no-locale
１２．設定ファイルの修正
まずは、/usr/local/pgsql/data/postgresql.conf を以下のように修正
log_destination = 'syslog'
log_statement = 'all'
listen_addresses = '*'
次に、/usr/local/pgsql/data/pg_hba.conf を以下のように修正
host all all 192.168.0.0/24 trust
参考サイト
syslogの設定なども上記サイトのように修正。
１３．Postgresqlの起動
$ pg_ctl -w start
１４．Postgresqlの起動確認
$ ps ax | grep postgres
１５．Postgresqlの停止
$ pg_ctl stop
１６．サーバマシンが起動する時に自動的にPostgreSQLが起動する設定にする。
→これは、postgresユーザーではなく、rootで行う。
# cd /etc/rc.d/init.d/
# cp -p /usr/local/src/postgresql-8.1.2/contrib/start-scripts/linux postgresql
# chown root.root postgresql
# chmod 755 postgresql
# chkconfig --level 2345 postgresql on
