In my previous blog I added some Python code. But when I published my code all code was aligned to the left-hand side. Unfortunately indentation is very important for Python so some googling got me to this site:
http://www.simplebits.com/cgi-bin/simplecode.pl
Add your code and click Process and then remove all br tags (e.g. by using vim)
Copy this result in the html version of the blog.
Lobo Loco - How about a nice big cup of ...
MV
Thursday, November 26, 2009
Python File Manipulation
Last week I ripped one of my dvd movies to have a backup. Unfortunately I forgot to add the dutch subtitles but I was too lazy to start all over again. So a simple google search gave me the dutch subtitle files, unfortunately not in sync with my dvd rip.
The subs appeared 1 min and something too fast, so my first intention was to manually edit the file... to no avail, after 2 or 3 lines I already gave up. This was going to take too much time: 1536 subtitles, appx 10s per subtitle... over 4 hours of work.
Ok, what could I do to get the subs synced? I think it's fairly easy by using bash scripting in combination with sed and awk, but I have no experience in bash, so that solution was out of the question.
What else could I do to get this file updated, and that's where Python came to the foreground, the only programming language I know a little, but enough to give it a try. The code in this blog has not indentation, so if you want to use this code, you only need to adjust the correct indentation.
The structure of the file looks like:
1518
02:26:23,091 --> 02:26:24,978
subtitle here
A first thing was to detect the lines that contain the start and end time of the subtitle. In the example above it's line 2 but of course I can not select the time lines by line number. The detection however, is fairly easy to do by using a regular expression.
This is the regex I use: ^(0[0-2])
The mechanism I use to update the file is as follows:
1. Read the source file
2. Check if line match the regular expression
2.1. if match: update the line by adding time to both time-stamps and write to an output file
2.2. if no match: write the line to the same output file.
Fairly straight-forward, isn't it?
When a line matches the regex, I need to filter the start and end time. Luckily these times are always in the same positions so I sliced the lines to get start and end time.
Once I have those times I do some calculations to calculate the new time and then I write the file to the new output file.
So in this file section I use the addTime function. This function expects a string
and will return a new string. This string is always of the format hh:mm:ss,xxx where xxx are the milliseconds. Unfortunately the separator isn't always the colon, so I have to slice up the input string to get the hour, minutes, seconds and milliseconds. Then I add to milliseconds, seconds and minutes the necessary time
and check if adding these times don't pass the normal hour times, since 64s does not exist in the real world. For example if number of seconds exceeds 59s, I add 1 to number of minutes and substract 60 from the number of seconds.
In the code below the "add*" variables are constants, defined somewhere else, so you can update to your own needs.
The updateTime function makes that all hh, mm and ss are always 2 digits and milli is always 3 digits.
We're there, I'm quite sure that this script could be done in less code by an experienced programmer, which I'm not. But at least it worked for me and it took me less than 1 hour to get this working, so it gave me time to write this blog and even then I would still be changing the file manually and probably with more mistakes.
Final code:
The subs appeared 1 min and something too fast, so my first intention was to manually edit the file... to no avail, after 2 or 3 lines I already gave up. This was going to take too much time: 1536 subtitles, appx 10s per subtitle... over 4 hours of work.
Ok, what could I do to get the subs synced? I think it's fairly easy by using bash scripting in combination with sed and awk, but I have no experience in bash, so that solution was out of the question.
What else could I do to get this file updated, and that's where Python came to the foreground, the only programming language I know a little, but enough to give it a try. The code in this blog has not indentation, so if you want to use this code, you only need to adjust the correct indentation.
The structure of the file looks like:
1518
02:26:23,091 --> 02:26:24,978
subtitle here
A first thing was to detect the lines that contain the start and end time of the subtitle. In the example above it's line 2 but of course I can not select the time lines by line number. The detection however, is fairly easy to do by using a regular expression.
This is the regex I use: ^(0[0-2])
The mechanism I use to update the file is as follows:
1. Read the source file
2. Check if line match the regular expression
2.1. if match: update the line by adding time to both time-stamps and write to an output file
2.2. if no match: write the line to the same output file.
Fairly straight-forward, isn't it?
When a line matches the regex, I need to filter the start and end time. Luckily these times are always in the same positions so I sliced the lines to get start and end time.
Once I have those times I do some calculations to calculate the new time and then I write the file to the new output file.
finput = open("/home/input", 'r')
foutput = open("/home/newsubtitles", 'w') #new empty file
for line in finput:
if re.match(regex, line):
start = line[0:12]
end = line[17:29]
newstart = addTime(start)
newend = addTime(end)
#replace start and end by newstart and newend
line.replace(start, newstart)
line.replace(end, newend)
#write updated line to output file
foutput.write(line)
else:
#in case of no match, just write line to output
foutput.write(line)
#when all lines are copied, close file
foutput.close()
So in this file section I use the addTime function. This function expects a string
and will return a new string. This string is always of the format hh:mm:ss,xxx where xxx are the milliseconds. Unfortunately the separator isn't always the colon, so I have to slice up the input string to get the hour, minutes, seconds and milliseconds. Then I add to milliseconds, seconds and minutes the necessary time
and check if adding these times don't pass the normal hour times, since 64s does not exist in the real world. For example if number of seconds exceeds 59s, I add 1 to number of minutes and substract 60 from the number of seconds.
In the code below the "add*" variables are constants, defined somewhere else, so you can update to your own needs.
def addTime(time):
hh = int(time[0:2])
mm = int(time[3:5])
ss = int(time[6:8])
milli = int(time[9:12])
milli = milli + addmilli
ss = ss + addss
mm = mm + addmm
if milli > 999:
milli = milli - 1000
ss = ss + 1
if ss > 59:
ss = ss - 60
mm = mm + 1
if mm > 59:
mm = mm - 60
hh = hh + 1
newtime = updateTime(hh, mm, ss, milli)
return newtime
The updateTime function makes that all hh, mm and ss are always 2 digits and milli is always 3 digits.
def updateTime(hh, mm, ss, milli):
milli = str(milli)
if len(milli) == 1:
milli = "00%s" %milli
elif len(milli) == 2:
milli = "0%s" %milli
ss = str(ss)
if len(ss) == 1:
ss = "0%s" %ss
mm = str(mm)
if len(mm) == 1:
mm = "0%s" %mm
hh = str(hh)
if len(hh) == 1:
hh = "0%s" %hh
newtime = "%s:%s:%s,%s" %(hh, mm, ss, milli)
return newtime
We're there, I'm quite sure that this script could be done in less code by an experienced programmer, which I'm not. But at least it worked for me and it took me less than 1 hour to get this working, so it gave me time to write this blog and even then I would still be changing the file manually and probably with more mistakes.
Final code:
import re
regex = "^(0[0-2])"
input = "/home/dewolfth/input.txt"
subs = "/home/dewolfth/mymovie.txt"
addhour = 1
addmm = 1
addss = 7
addmilli = 500
def updateTime(hh, mm, ss, milli):
milli = str(milli)
if len(milli) == 1:
milli = "00%s" %milli
elif len(milli) == 2:
milli = "0%s" %milli
ss = str(ss)
if len(ss) == 1:
ss = "0%s" %ss
mm = str(mm)
if len(mm) == 1:
mm = "0%s" %mm
hh = str(hh)
if len(hh) == 1:
hh = "0%s" %hh
newtime = "%s:%s:%s,%s" %(hh, mm, ss, milli)
return newtime
def addTime(time):
hh = int(time[0:2])
mm = int(time[3:5])
ss = int(time[6:8])
milli = int(time[9:12])
milli = milli + addmilli
ss = ss + addss
mm = mm + addmm
if milli > 999:
milli = milli - 1000
ss = ss + 1
if ss > 59:
ss = ss - 60
mm = mm + 1
if mm > 59:
mm = mm - 60
hh = hh + 1
newtime = updateTime(hh, mm, ss, milli)
return newtime
def main():
finput = open(input, 'r')
foutput = open(subs, 'w')
for line in finput:
if re.match(regex,line):
start = line[0:12]
newstart = addTime(start)
end = line[17:29]
newend = addTime(end)
line = line.replace(start, newstart)
line = line.replace(end, newend)
foutput.write(line)
else:
foutput.write(line)
foutput.close()
if __name__ == "__main__":
main()
Wednesday, November 25, 2009
From techn writer to software developer and back and ...
The last weeks I have been lucky to do some development. Even though the tasks I did, would be for a sr developer some minor work, the job helped me getting a better view over our software. All development I did was in Python.
My first job was to refactor a wizard so that it became consistent to the new specs of the software.
After this refactoring work, I refactored the same wizard again and two others so that these wizards became available in the web interface.
A lot of development and testing later, I succeeded in all my tasks, I even did some bug-fixes, w00t.
But time has come that my engineering tasks are over, and that I'm back in my natural habitat of writing documentation. I'm a tech writer after all...
So what did I learn during these last weeks as developer?
First I have to admit that I really enjoyed my time as engineer. Not that I've become an experienced engineer (pretty far away from that level), but I just like to learn new stuff. And just like everything, you learn by doing so. So I'm glad that my manager gave me the opportunity to spend some time in development, since I don't have too much time in private to become a developer.
I have learned quite a lot about Q-layer's Q-Action/Workflow mechanism and using variables in that system. Even though it's quite complex, imho it's a great system.
Once you get to know the terminology and mechanism, it goes quite fast to create your own Q-Actions, annex workflows.
So before returning completely to tech writing mode, I was thinking, why not write a short message before I start documenting the things I learned the passed few weeks...
Bye bye til the next time!
My first job was to refactor a wizard so that it became consistent to the new specs of the software.
After this refactoring work, I refactored the same wizard again and two others so that these wizards became available in the web interface.
A lot of development and testing later, I succeeded in all my tasks, I even did some bug-fixes, w00t.
But time has come that my engineering tasks are over, and that I'm back in my natural habitat of writing documentation. I'm a tech writer after all...
So what did I learn during these last weeks as developer?
First I have to admit that I really enjoyed my time as engineer. Not that I've become an experienced engineer (pretty far away from that level), but I just like to learn new stuff. And just like everything, you learn by doing so. So I'm glad that my manager gave me the opportunity to spend some time in development, since I don't have too much time in private to become a developer.
I have learned quite a lot about Q-layer's Q-Action/Workflow mechanism and using variables in that system. Even though it's quite complex, imho it's a great system.
Once you get to know the terminology and mechanism, it goes quite fast to create your own Q-Actions, annex workflows.
So before returning completely to tech writing mode, I was thinking, why not write a short message before I start documenting the things I learned the passed few weeks...
Bye bye til the next time!
Thursday, September 10, 2009
Reading manuals is not wasting time... sometimes
I wanted to test Oracle's Enterprise Linux (Unbreakable Linux). On the website of Oracle I couldn't find one big iso, so I downloaded the 5 CD iso images.
As I don't have enough computers, I use VirtualBox. Since I thought that it was not possible to install an OS via multiple iso images with VirtualBox, I did the following:
I merged the 5 iso images into one big iso image. Don't worry, this doesn't require complex manipulations, this is what I did:
1. copied the first iso to a new iso image: "cp first.iso new.iso"
2. executed the next command: "cat second.iso >> new.iso"
This just appends the second iso image to the copy of the first iso image.
Repeat this for the other iso images ("cat third.iso >> new.iso", "cat fourth.iso >> new.iso", and "cat fifth.iso >> new.iso")
Unfortunately, during the installation of Enterprise Linux, the wizard kept asking for Installation CD 2... No luck for me.
Back to installing via the 5 iso images then. And after a tip I found out that it is easy to switch from iso image during the installation.
First make sure that all iso images are loaded in VirtualBox.
Then create the a virtual machine and let it boot from the first iso image.
When the install wizards asks for the second iso image, click Devices in the VirtualBox menu, and select Unmount CD/DVD-ROM.
Then Devices > Mount CD/DVD-ROM > CD/DVD-ROM Image and select the second iso image.
Simple as that... perhaps I should try reading manuals...
As I don't have enough computers, I use VirtualBox. Since I thought that it was not possible to install an OS via multiple iso images with VirtualBox, I did the following:
I merged the 5 iso images into one big iso image. Don't worry, this doesn't require complex manipulations, this is what I did:
1. copied the first iso to a new iso image: "cp first.iso new.iso"
2. executed the next command: "cat second.iso >> new.iso"
This just appends the second iso image to the copy of the first iso image.
Repeat this for the other iso images ("cat third.iso >> new.iso", "cat fourth.iso >> new.iso", and "cat fifth.iso >> new.iso")
Unfortunately, during the installation of Enterprise Linux, the wizard kept asking for Installation CD 2... No luck for me.
Back to installing via the 5 iso images then. And after a tip I found out that it is easy to switch from iso image during the installation.
First make sure that all iso images are loaded in VirtualBox.
Then create the a virtual machine and let it boot from the first iso image.
When the install wizards asks for the second iso image, click Devices in the VirtualBox menu, and select Unmount CD/DVD-ROM.
Then Devices > Mount CD/DVD-ROM > CD/DVD-ROM Image and select the second iso image.
Simple as that... perhaps I should try reading manuals...
Thursday, September 3, 2009
Turn Ubuntu Jaunty into Mac OSX Leopard
Just to test if mac layout is so much better than gnome, well, not really, imho they're equal, though, I must admit that mac layout is just that bit fancier...
On this site it is really good explained how you can turn your ubuntu (intrepid) into mac osx leopard layout. This tutorial has been tested on Ubuntu Jaunty
The only part where I didn't follow the tutorial, is in the section Configuring usplash screen where you need to download the .deb files (splashy_....deb and libsplashy....deb)
I had to execute this command to get splashy working:
dpkg –force-overwrite -i /var/cache/apt/archives/splashy_0.3.13-3ubuntu1_i386.deb
Hope to get a real mac soon ...
On this site it is really good explained how you can turn your ubuntu (intrepid) into mac osx leopard layout. This tutorial has been tested on Ubuntu Jaunty
The only part where I didn't follow the tutorial, is in the section Configuring usplash screen where you need to download the .deb files (splashy_....deb and libsplashy....deb)
I had to execute this command to get splashy working:
dpkg –force-overwrite -i /var/cache/apt/archives/splashy_0.3.13-3ubuntu1_i386.deb
Hope to get a real mac soon ...
Wednesday, September 2, 2009
Get to know the power of GIMP
I recently bought me a DSLR (Sony Alpha-300). Of course the images, taken by my new dslr are too big to publish on websites, such as picasaweb.
I used to open my windows in VirtualBox and then use Microsoft's Picture Manager (comes with Office) to resize my images to web-quality. I have to admit that this worked fine, but I got annoyed by the slow access speeds from my virtual windows to my local drives.
I heard that GIMP could be used as batch system, so I started looking around to see how I could compress my images to web-quality for multiple files. And in fact, it's fairly easy. GIMP comes with the most common Linux distributions, you only need to install gimp-plugin-registry with your package manager (apt-get, yast, yum, ...).
Once the package is installed, start GIMP. The batch processor can be found under Filters > Batch > Batch Process.
In the window that appear you can select your images, and the actions to be executed on each tab. For example for compressing images, go to the Output tab, select the image type and select a quality level.
And yet another step away from Windows :-)
I used to open my windows in VirtualBox and then use Microsoft's Picture Manager (comes with Office) to resize my images to web-quality. I have to admit that this worked fine, but I got annoyed by the slow access speeds from my virtual windows to my local drives.
I heard that GIMP could be used as batch system, so I started looking around to see how I could compress my images to web-quality for multiple files. And in fact, it's fairly easy. GIMP comes with the most common Linux distributions, you only need to install gimp-plugin-registry with your package manager (apt-get, yast, yum, ...).
Once the package is installed, start GIMP. The batch processor can be found under Filters > Batch > Batch Process.
In the window that appear you can select your images, and the actions to be executed on each tab. For example for compressing images, go to the Output tab, select the image type and select a quality level.
And yet another step away from Windows :-)
Friday, August 28, 2009
Numerical keypad doesn't work anymore
Every once in a while, my numerical keypad only works as a mouse on my Ubuntu. I'm experiencing this behavior since the Intrepid version (8.10). Luckily for me it's not a big deal to solve it, but I had to google it every time. So perhaps by writing this small blog, it stays in my mind.
So to get the numerical keypad working again, go to System > Preferences > Keyboard
Then go to the tab Mouse Keys and clear the option "Pointer can be controlled using the keypad".
I still need to figure out which update causes this behavior...
My thanks to the person of this solution (http://bit.ly/47sAQ).
Cheerio!
So to get the numerical keypad working again, go to System > Preferences > Keyboard
Then go to the tab Mouse Keys and clear the option "Pointer can be controlled using the keypad".
I still need to figure out which update causes this behavior...
My thanks to the person of this solution (http://bit.ly/47sAQ).
Cheerio!
Friday, July 17, 2009
Clustering ejabberd nodes using mnesia databases
A while ago, I was asked if I could set up a cluster of 2 ejabberd nodes (a Jabber/XMPP instant messaging server), using the default internal mnesia database. I found this a nice challenge, which shouldn't take too much time... NOT!
I use 2 servers, both running OpenSolaris on which I have installed a very nice framework, namely PyMonkey (see www.pymonkey.org and www.qshell.org), used for building compute and storage clouds amongst others. To work on this framework I need to be root in "/opt", in which the framework is installed. I installed the ejabberd (v. 2.0.5) and erlang (v. R12B-5) packages in this framework (Solaris only, as we speak). It's not that I'm working in a specific framework that this information is not valid for the default packages on Linux, Windows, or (Open)Solaris. Differences might be the location of the mnesia database.
So I started my journey in googling for information on the net about installing ejabberd nodes. Quickly I got to the official ejabberd guide, which is clear and a good starting point (1). I soon found out that setting up one node is pretty straight-forward, setting up nodes in a cluster however is another piece of cake. Looking further in the official guide I even got the documentation of clustering right in front of me (2). However, what is described in that chapter is far from accurate to get a cluster of ejabberd nodes.
Some more googling lead me to various blogs (3) and internet fora (4)(5), some of them more helpful than the other, but they all helped me a little further and I finally got my nodes into clustering, thanks to my colleague, who gave me the last hint.
The big issue was the location of the cookie (.erlang.cookie) on the new ejabberd node. The cookie is required to identify the different nodes in the cluster. This cookie must be identical on all nodes, otherwise the nodes won't be able to communicate with each other. Some mailings (6) pointed also to ".hosts.erlang" as cause for my issue, but in my test case I didn't need that file.
To save you a lot of time I decided to share my procedure in my blog.
Here is my setup:
* OpenSolaris ejabberd1
* OpenSolaris ejabberd2
* Both running the PyMonkey framework (/opt/qbase3). Download the PyMonkey sandbox on http://confluence.qlayer.com/display/SOFT/Download
* I can ping from one machine to the other by using the host name. Possibly you have to change this in /etc/hosts
On both machines:
* the ejabberd package is installed in /opt/qbase3/apps/ejabberd-2.0.5, further noted as $EJABBERD
* the erlang/otp package is installed in /opt/qbase3/apps/erlang-R12B-5, further noted as $ERLANG
* I am logged on as root
Set up First Node (on ejabberd1)
1. Create backups of some critical files (just in case...):
* ejabberd.cfg in $EJABBERD/etc/ejabberd
* ejabberdctl in $EJABBERD/sbin/
2. Modify ejabberd.cfg
In the section SERVED HOSTNAMES I have a line "{hosts, ["localhost"]}.". (!) I'm careful not to remove any periods at the end of lines.
I change localhost by a domain (e.g. myexample.com), served by my first ejabberd node, as clearly stated in the example in the above lines of the file.
In the section ACCESS CONTROL LISTS I have 2 commented examples of users who will have admin rights on the ejabberd node, for example "%%{acl, admin, {user, "aleksey", "localhost"}}."
I remove the %-signs (which indicate comments) and then change "aleksey" and "localhost" by my username (e.g. tdewolf) and my domain (the one defined as served hostname).
Remark:
If I want to create multiple admin users, I have to create a line for each admin user.
I will still need to register the users to the ejabberd node however once the node is up and running, see later.
3. Modify ejabberdctl
At approximately line 12, I find the line "HOST=localhost", where I can set the host name of my ejabberd node. This is the name of the host of the ejabberd node name. Use the server's hostname, ejabberd1.
Remind that this host is not the same as host in the ejabberd.cfg file!
At the last line of that section ("EJABBERD_DB=$ROOTDIR/var/lib/ejabberd/db/$NODE"), approximately line 18, I change $NODE to $ERLANG_NODE, to obtain a clearer name for the database.
4. Since I'm working in a sandbox, I also need to set my environment variable LD_LIBRARY_PATH to point to /opt/qbase3/lib.
Remark: I could omit this step if I would use the default ejabberd and erlang packages.
5. Ok, now I'm ready to launch my first ejabberd node.
From $EJABBERD, I enter the following command: ./sbin/ejabberdctl start
My node should be started, so I check the status by issuing: ./sbin/ejabberdctl status
It returns:
Node ejabberd@ejabberd1 is started. Status: started
ejabberd is running
6. I will now register the admin users that I have set in step 2, issuing the command: ./sbin/ejabberdctl register user host password
where:
*user: the user name as defined in ejabberd.cfg
*host: the host name as defined in ejabberd.cfg
*password: a randomly chosen password
With these credentials you are able to open ejabberd's WebAdmin at http://ip node:5280/admin/. Use the full Jabber ID to log on, for example tdewolf@myexample.com.
Visit the official guide for more information about the WebAdmin (7). I leave the WebAdmin session open to easily see a new ejabberd node appear, therefore I go to the Nodes page.
I have an ejabberd node up and running and I can even access the WebAdmin of it, w00t. That was easy, wasn't it?
When the ejabberd node has started the following files and directories are created:
* $EJABBERD/var/lib/ejabberd/.erlang.cookie
* $EJABBERD/var/lib/ejabberd/db/ejabberd@myexample.com (host is defined in ejabberdctl)
Set up Second Node (on ejabberd2)
1. Copy the necessary files from your first node to my second node.
Copy from the first node $EJABBERD/var/lib/ejabberd/.erlang.cookie to /root on this new node and to $EJABBERD/var/lib/ejabberd. The default working directory of erlang is $HOME, and since I need to be root to be able to work in my sandbox, /root is my $HOME.
Copy from the first node $EJABBERD/etc/ejabberd/ejabberd.cfg to the same location on this new node. This means that this new ejabberd node will serve the same domain and that it will have the same admin users.
2. Modify ejabberdctl
At approximately line 12, I find the line "HOST=localhost", where I can set the host name of my ejabberd node. This is the name of the host of the ejabberd node name. Use the server's hostname, ejabberd2.
Remind that this host is not the same as host in the ejabberd.cfg file!
At the last line of that section ("EJABBERD_DB=$ROOTDIR/var/lib/ejabberd/db/$NODE"), approximately line 18, I change $NODE to $ERLANG_NODE, to obtain a clearer name for the database.
3. Since I'm working in a sandbox, I also need to set my environment variable LD_LIBRARY_PATH to point to /opt/qbase3/lib.
Remark: I could omit this step if I would use the default ejabberd and erlang packages.
4. Now I'm ready to set my new ejabberd node in cluster with the first node. From $ERLANG/bin I enter the following very long command.
(!) Take care about ALL quotes and omit the backslashes, they just indicate to continue the command:
./erl -sname ejabberd@ejabberd2 \
-mnesia dir '"/opt/qbase3/apps/ejabberd-2.0.5/var/lib/ejabberd/db/"' \
-mnesia extra_db_nodes "['ejabberd@ejabberd1']" \
-s mnesia
where:
* -sname ejabberd@ejabberd2 is the new ejabberd node
* -mnesia dir '"..."' is the location of the database on the new node, by default in var/lib/ejabberd/db/. The outer quotes is a single quote; the inner pair is a double quote!
* -mnesia extra_db_nodes "['...']" is the node to make the cluster with, mind again the single and double quotes!
5. I arrive in an erlang shell when I have executed the above command: (ejabberd@ejabberd2)1>
To see if my two nodes are in a cluster I enter the command "mnesia:info()." then scroll up to find a line "running db nodes" which must be a list with the two nodes.
Double-check: I see the second ejabberd node appear under Running Nodes in the WebAdmin of my first ejabberd node, w00t again!
Once again, mind the period at the end of the command!
***Tip:
If you find only one ejabberd node in running db nodes and the other one in stopped db nodes, something went wrong.
Most likely your .erlang.cookie is located in the wrong location. In your erlang session, execute "erlang:get_cookie()." and note the result between the quotes, e.g. IXHDCSATUADBDTKVTOFC. Then lookup .erlang.cookie with that result and overwrite it with the version from ejabberd node 1.
***
6. Synchronize databases
I need to synchronize the database of my new node with the database of my first node. At the Erlang Shell, I enter the following command:
mnesia:change_table_copy_type(schema, node(), disc_copies).
Quit the Erlang Shell with the command "q()." The second ejabberd node is now listed in Stopped Nodes! Check in a refreshed Nodes page in the first node's WebAdmin.
7. Now that I know that my two nodes can work in a cluster, I need to change $EJABBERD/sbin/ejabberdctl to run with the same parameters as I did with the command in step 4.
In the section "# start server" insert a new line after the line "-mnesia dir "\"$EJABBERD_DB\"" \". Add the following content to this new line (without the start and end quote): "-mnesia extra_db_nodes "['ejabberd@ejabberd1']" -s mnesia \"
which are the last two parameters of the .erl command.
***Tip:
If you would use the node in "live" mode, do the same in the section "# start interactive server"
***
8. It's time now to start my second node in the normal way. From $EJABBERD I enter the command:
./sbin/ejabberdctl start
In WebAdmin of the first node, I see my second node appear in Running Nodes, which means that both ejabberd nodes are running and active in one cluster.
To get the information written to both databases, I change the storage type of a database (Nodes > select node > Database) and click Submit at the bottom of the page to apply my changes.
For more information about the parameters and functions I can advise you to visit the official ejabberd guide (1) and the official erlang site (8).
A good tutorial for a Linux setup can be found on sysmonblog (9).
(1) http://www.process-one.net/en/ejabberd/guide_en
(2) http://www.process-one.net/en/ejabberd/guide_en#htoc79
(3) http://dev.esl.eu/blog/2008/09/30/set-up-clustering-in-ejabberd/
(4) http://www.nabble.com/problems-in-clustering-ejabberd-via-tutorial-on-official-site-td22475081.html
(5) http://www.trapexit.org/forum/viewtopic.php?p=43930
(6) http://lists.jabber.ru/pipermail/ejabberd/2005-March/000883.html
(7) http://www.process-one.net/en/ejabberd/guide_en#webadmin
(8) http://erlang.org/
(9) http://sysmonblog.co.uk/2008/06/ot-installing-ejabberd-on-debian-ubuntu.html
I use 2 servers, both running OpenSolaris on which I have installed a very nice framework, namely PyMonkey (see www.pymonkey.org and www.qshell.org), used for building compute and storage clouds amongst others. To work on this framework I need to be root in "/opt", in which the framework is installed. I installed the ejabberd (v. 2.0.5) and erlang (v. R12B-5) packages in this framework (Solaris only, as we speak). It's not that I'm working in a specific framework that this information is not valid for the default packages on Linux, Windows, or (Open)Solaris. Differences might be the location of the mnesia database.
So I started my journey in googling for information on the net about installing ejabberd nodes. Quickly I got to the official ejabberd guide, which is clear and a good starting point (1). I soon found out that setting up one node is pretty straight-forward, setting up nodes in a cluster however is another piece of cake. Looking further in the official guide I even got the documentation of clustering right in front of me (2). However, what is described in that chapter is far from accurate to get a cluster of ejabberd nodes.
Some more googling lead me to various blogs (3) and internet fora (4)(5), some of them more helpful than the other, but they all helped me a little further and I finally got my nodes into clustering, thanks to my colleague, who gave me the last hint.
The big issue was the location of the cookie (.erlang.cookie) on the new ejabberd node. The cookie is required to identify the different nodes in the cluster. This cookie must be identical on all nodes, otherwise the nodes won't be able to communicate with each other. Some mailings (6) pointed also to ".hosts.erlang" as cause for my issue, but in my test case I didn't need that file.
To save you a lot of time I decided to share my procedure in my blog.
Here is my setup:
* OpenSolaris ejabberd1
* OpenSolaris ejabberd2
* Both running the PyMonkey framework (/opt/qbase3). Download the PyMonkey sandbox on http://confluence.qlayer.com/display/SOFT/Download
* I can ping from one machine to the other by using the host name. Possibly you have to change this in /etc/hosts
On both machines:
* the ejabberd package is installed in /opt/qbase3/apps/ejabberd-2.0.5, further noted as $EJABBERD
* the erlang/otp package is installed in /opt/qbase3/apps/erlang-R12B-5, further noted as $ERLANG
* I am logged on as root
Set up First Node (on ejabberd1)
1. Create backups of some critical files (just in case...):
* ejabberd.cfg in $EJABBERD/etc/ejabberd
* ejabberdctl in $EJABBERD/sbin/
2. Modify ejabberd.cfg
In the section SERVED HOSTNAMES I have a line "{hosts, ["localhost"]}.". (!) I'm careful not to remove any periods at the end of lines.
I change localhost by a domain (e.g. myexample.com), served by my first ejabberd node, as clearly stated in the example in the above lines of the file.
In the section ACCESS CONTROL LISTS I have 2 commented examples of users who will have admin rights on the ejabberd node, for example "%%{acl, admin, {user, "aleksey", "localhost"}}."
I remove the %-signs (which indicate comments) and then change "aleksey" and "localhost" by my username (e.g. tdewolf) and my domain (the one defined as served hostname).
Remark:
If I want to create multiple admin users, I have to create a line for each admin user.
I will still need to register the users to the ejabberd node however once the node is up and running, see later.
3. Modify ejabberdctl
At approximately line 12, I find the line "HOST=localhost", where I can set the host name of my ejabberd node. This is the name of the host of the ejabberd node name. Use the server's hostname, ejabberd1.
Remind that this host is not the same as host in the ejabberd.cfg file!
At the last line of that section ("EJABBERD_DB=$ROOTDIR/var/lib/ejabberd/db/$NODE"), approximately line 18, I change $NODE to $ERLANG_NODE, to obtain a clearer name for the database.
4. Since I'm working in a sandbox, I also need to set my environment variable LD_LIBRARY_PATH to point to /opt/qbase3/lib.
Remark: I could omit this step if I would use the default ejabberd and erlang packages.
5. Ok, now I'm ready to launch my first ejabberd node.
From $EJABBERD, I enter the following command: ./sbin/ejabberdctl start
My node should be started, so I check the status by issuing: ./sbin/ejabberdctl status
It returns:
Node ejabberd@ejabberd1 is started. Status: started
ejabberd is running
6. I will now register the admin users that I have set in step 2, issuing the command: ./sbin/ejabberdctl register
where:
*
*
*
With these credentials you are able to open ejabberd's WebAdmin at http://
Visit the official guide for more information about the WebAdmin (7). I leave the WebAdmin session open to easily see a new ejabberd node appear, therefore I go to the Nodes page.
I have an ejabberd node up and running and I can even access the WebAdmin of it, w00t. That was easy, wasn't it?
When the ejabberd node has started the following files and directories are created:
* $EJABBERD/var/lib/ejabberd/.erlang.cookie
* $EJABBERD/var/lib/ejabberd/db/ejabberd@myexample.com (host is defined in ejabberdctl)
Set up Second Node (on ejabberd2)
1. Copy the necessary files from your first node to my second node.
Copy from the first node $EJABBERD/var/lib/ejabberd/.erlang.cookie to /root on this new node and to $EJABBERD/var/lib/ejabberd. The default working directory of erlang is $HOME, and since I need to be root to be able to work in my sandbox, /root is my $HOME.
Copy from the first node $EJABBERD/etc/ejabberd/ejabberd.cfg to the same location on this new node. This means that this new ejabberd node will serve the same domain and that it will have the same admin users.
2. Modify ejabberdctl
At approximately line 12, I find the line "HOST=localhost", where I can set the host name of my ejabberd node. This is the name of the host of the ejabberd node name. Use the server's hostname, ejabberd2.
Remind that this host is not the same as host in the ejabberd.cfg file!
At the last line of that section ("EJABBERD_DB=$ROOTDIR/var/lib/ejabberd/db/$NODE"), approximately line 18, I change $NODE to $ERLANG_NODE, to obtain a clearer name for the database.
3. Since I'm working in a sandbox, I also need to set my environment variable LD_LIBRARY_PATH to point to /opt/qbase3/lib.
Remark: I could omit this step if I would use the default ejabberd and erlang packages.
4. Now I'm ready to set my new ejabberd node in cluster with the first node. From $ERLANG/bin I enter the following very long command.
(!) Take care about ALL quotes and omit the backslashes, they just indicate to continue the command:
./erl -sname ejabberd@ejabberd2 \
-mnesia dir '"/opt/qbase3/apps/ejabberd-2.0.5/var/lib/ejabberd/db/"' \
-mnesia extra_db_nodes "['ejabberd@ejabberd1']" \
-s mnesia
where:
* -sname ejabberd@ejabberd2 is the new ejabberd node
* -mnesia dir '"..."' is the location of the database on the new node, by default in var/lib/ejabberd/db/. The outer quotes is a single quote; the inner pair is a double quote!
* -mnesia extra_db_nodes "['...']" is the node to make the cluster with, mind again the single and double quotes!
5. I arrive in an erlang shell when I have executed the above command: (ejabberd@ejabberd2)1>
To see if my two nodes are in a cluster I enter the command "mnesia:info()." then scroll up to find a line "running db nodes" which must be a list with the two nodes.
Double-check: I see the second ejabberd node appear under Running Nodes in the WebAdmin of my first ejabberd node, w00t again!
Once again, mind the period at the end of the command!
***Tip:
If you find only one ejabberd node in running db nodes and the other one in stopped db nodes, something went wrong.
Most likely your .erlang.cookie is located in the wrong location. In your erlang session, execute "erlang:get_cookie()." and note the result between the quotes, e.g. IXHDCSATUADBDTKVTOFC. Then lookup .erlang.cookie with that result and overwrite it with the version from ejabberd node 1.
***
6. Synchronize databases
I need to synchronize the database of my new node with the database of my first node. At the Erlang Shell, I enter the following command:
mnesia:change_table_copy_type(schema, node(), disc_copies).
Quit the Erlang Shell with the command "q()." The second ejabberd node is now listed in Stopped Nodes! Check in a refreshed Nodes page in the first node's WebAdmin.
7. Now that I know that my two nodes can work in a cluster, I need to change $EJABBERD/sbin/ejabberdctl to run with the same parameters as I did with the command in step 4.
In the section "# start server" insert a new line after the line "-mnesia dir "\"$EJABBERD_DB\"" \". Add the following content to this new line (without the start and end quote): "-mnesia extra_db_nodes "['ejabberd@ejabberd1']" -s mnesia \"
which are the last two parameters of the .erl command.
***Tip:
If you would use the node in "live" mode, do the same in the section "# start interactive server"
***
8. It's time now to start my second node in the normal way. From $EJABBERD I enter the command:
./sbin/ejabberdctl start
In WebAdmin of the first node, I see my second node appear in Running Nodes, which means that both ejabberd nodes are running and active in one cluster.
To get the information written to both databases, I change the storage type of a database (Nodes > select node > Database) and click Submit at the bottom of the page to apply my changes.
For more information about the parameters and functions I can advise you to visit the official ejabberd guide (1) and the official erlang site (8).
A good tutorial for a Linux setup can be found on sysmonblog (9).
(1) http://www.process-one.net/en/ejabberd/guide_en
(2) http://www.process-one.net/en/ejabberd/guide_en#htoc79
(3) http://dev.esl.eu/blog/2008/09/30/set-up-clustering-in-ejabberd/
(4) http://www.nabble.com/problems-in-clustering-ejabberd-via-tutorial-on-official-site-td22475081.html
(5) http://www.trapexit.org/forum/viewtopic.php?p=43930
(6) http://lists.jabber.ru/pipermail/ejabberd/2005-March/000883.html
(7) http://www.process-one.net/en/ejabberd/guide_en#webadmin
(8) http://erlang.org/
(9) http://sysmonblog.co.uk/2008/06/ot-installing-ejabberd-on-debian-ubuntu.html
Monday, July 13, 2009
Ubuntu installation on recent hardware
Last January I bought a new laptop (HP DV7), and of course Windows Vista was included. My first actions on my new laptop was reinstalling the machine with Ubuntu.
The installation is easy and straight forward, except for getting sound out of the machine (Intel's ICH9 chipset). After googling and exploring ubuntu, I finally got it working.
Here's my installation procedure:
The installation is easy and straight forward, except for getting sound out of the machine (Intel's ICH9 chipset). After googling and exploring ubuntu, I finally got it working.
Here's my installation procedure:
- Install Ubuntu
- When installed update and upgrade your complete installation:
sudo apt-get update
sudo apt-get upgrade
Never mind any nvidia driver settings, do this afterwards.
Most likely you will have to perform this step twice. - Update the Hardware drivers: System > Administration > Hardware Drivers:
Use the Ubuntu recommended driver, enter password when asked for. - Sound might not work (stuttering, no sound, ...): add "irqpoll" to kernel command
- open "/boot/grub/menu.lst"
- look for the following lines:
title Ubuntu hardy (development branch), kernel 2.6.24-16-generic
root (hd0,5)
kernel /boot/vmlinuz-2.6.24-16-generic root=UUID=8k4o3ilj-3fff-dfae-943j-4332lkj52kl43 ro quiet splash
initrd /boot/initrd.img-2.6.24-16-generic - add "irqpoll" at the end of the line "kernel"
- save and reboot the pc
- Mic might not work:
- remove pulseaudio and install esound:
sudo apt-get remove pulseaudio
sudo apt-get install esound
sudo rm /etc/X11/Xsession.d/70pulseaudio (You may want to back this up) - Under System -> Preferences -> Sound make sure they are all set to 'Autodetect'.
The only one you will have to set manually to ALSA is 'Sound Capture' under 'Audio Conferencing'.
Note at this point Pulseaudio is now no longer an option under these drop menus. - Under System -> Preferences -> Sessions
Deselected or Remove the Pulseaudio Manager - Remove and asoundrc config file from your home dir (~/):
rm .asound*
- Remove pulseaudio packages:
sudo apt-get --purge remove pulseaudio
sudo apt-get autoremove
sudo apt-get remove pulseaudio-utils
sudo apt-get remove pulseaudio-module-conf
sudo apt-get remove pulseaudio-module-hal
sudo apt-get remove pulseaudio-module-x11
sudo apt-get remove libpulsecore5
sudo apt-get remove gstreamer0.10-pulseaudio
sudo apt-get remove libpulse-browse0 - Reboot
- Check Alsamixer settings, master may be reset to zero:
$ alsamixer -Dhw
- Install the following software:
- thunderbird
- skype (see medibuntu.org)
- compizconfig-settings-manager
- avant-window-navigator
- subversion
- banshee
- amarok
- acroread (see medibuntu.org)
- gnome-do
- eid card reader (see http://eid.belgium.be/nl/)
- k3b
- virtualbox (http://www.virtualbox.org/wiki/Linux_Downloads)
- gvim
- optional:network-manager-vpnc, xchat, bittorrent/deluge, gFTP
- openssh-server (see http://www.cyberciti.biz/faq/ubuntu-linux-openssh-server-installation-and-configuration/)
Thursday, June 18, 2009
daan
The catchy song Exes of Daan is stuck in my head. Got myself some time on youtube where I found this clip (in Dutch) where "De Rode Loper" (from Een, a Belgian TV Station) visits his house and recording studio.
(http://www.youtube.com/watch?v=Cgp9GE7Sw3c)
Daan Stuyven, keep bringing me joy in my life with your great songs, YOU RULE!
(http://www.youtube.com/watch?v=Cgp9GE7Sw3c)
Daan Stuyven, keep bringing me joy in my life with your great songs, YOU RULE!
Subscribe to:
Posts (Atom)