Public » Miscellaneous » Rediqueuelous A silly experiment with Redis-based, reloading queueing systems
Clone URL:  
Pushed to one repository · View In Graph Contained in tip

add per-machine queues and a config file

Changeset 017cf45fd214…

Parent 12a41256ba1c…

by Profile picture of Benjamin PollackBenjamin Pollack

Changes to 4 files · Browse files at 017cf45fd214 Showing diff from parent 12a41256ba1c Diff from another changeset...

Change 1 of 1 Show Entire File config.yaml.example Stacked
 
 
 
 
 
 
1
2
3
4
@@ -0,0 +1,4 @@
+name: default +redis: + host: localhost + port: 6379
Change 1 of 2 Show Entire File halp.txt Stacked
 
 
 
1
2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
4
5
 
22
23
24
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
38
39
40
41
42
43
44
45
46
@@ -1,5 +1,21 @@
+OVERVIEW +  Rediqueuelous is a stupidly simple Python queueing system.   +CONFIGURATION + +You don't need to configure anything if you only have one +Rediqueuelous server and you're running Redis locally. Otherwise, +create a file called config.yaml with values that look similar to +config.yaml.example, but are valid. + +If you have more than one Rediqueuelous processor running, you should +make sure each machine has a distinct "name" setting in its +config.yaml. This will allow you to enqueue items on specific +machines, if you wish to. + +USAGE +  To make a task happen, simply lpush a string in the format  "module.function" or "module.function:args", where "args" is a  JSON-encoded dictionary, into the queue called "rediqueue" in your @@ -22,3 +38,9 @@
 "tea_time", which takes a mod=modulename pair, and will load or reload  modulename. This allows you to redefine functions in Rediqueuelous  without restarting the server. + +ADVANCED + +Each machine also has a private queue, named "rediqueue!whatever", +where "whatever" is whatever you put into the name field in your +config.yaml. By default, it's "default".
Change 1 of 2 Show Entire File rediqueuelous.py Stacked
 
6
7
8
 
9
10
11
12
 
 
 
 
 
 
 
 
 
 
 
 
13
14
15
 
30
31
32
33
 
34
35
36
 
6
7
8
9
10
11
12
 
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 
42
43
44
 
45
46
47
48
@@ -6,10 +6,22 @@
   import redis  import simplejson +import yaml    RUN = True  MODULES = {'rediqueuelous': sys.modules[__name__]} -R = redis.Redis() +try: + _conf = yaml.load(open('config.yaml')) +except: + _conf = { + 'name': 'default', + 'redis': { + 'host': 'localhost', + 'port': '6379' + } + } +R = redis.Redis(_conf['redis']['host'], int(_conf['redis']['port'])) +NAME = _conf['name']    def quit():   global RUN @@ -30,7 +42,7 @@
 def process():   global RUN   while RUN: - job = R.brpop('rediqueue')[1] + job = R.brpop(['rediqueue', 'rediqueue!' + NAME])[1]   if ':' in job:   job, args = job.split(':', 1)   kwargs = simplejson.loads(args)
Change 1 of 1 Show Entire File requirements.txt Stacked
 
1
2
 
3
4
 
5
6
 
1
2
3
4
5
6
7
 
@@ -1,6 +1,7 @@
 Flask==0.6.1  Jinja2==2.5.5 +PyYAML==3.09  Werkzeug==0.6.2  redis==2.2.2 +simplejson>=2.0.0  wsgiref==0.1.2 -simplejson>=2.0.0