transcoding « WordPress.com Tag Feed

Soon It Will Be Cheaper to Compute Than to Cache

Stacey Higginbotham — Sat, 14 Nov 2009 01:05:21 +0000

At the NewTeeVee Live conference in San Francisco yesterday, I interviewed Sam Blackman, co-founder and CEO of Elemental Technologies, about trends on computing and storage costs. Elemental makes software that takes advantage of graphics processors to provide super fast transcoding, which is the process by which video content is formatted for different devices. The idea of storing one copy of a piece of content and formatting it on the fly makes intuitive sense when compared to storing four or five copies of that content and then delivering it only when needed.

But for now the cost of storing multiple copies is actually cheaper than doing the transcoding in real time. That difference in cost has been a barrier keeping some service providers from buying products that offer capabilities such as those of Elemental’s new server, which launched yesterday. However, Blackman believes that will change within the next three years as prices for processing fall faster than the price of storage. A similar trend is enabling people to store more of their data online, rather than on their hard drives. The cost of bandwidth is so low that folks don’t mind either streaming content or looking it up online rather than downloading it and keeping it on their computers. Check out Blackman’s comments in the video below.

Series: Ripping/Transcoding Tools In Linux – Acidrip

Ankit Sangwan — Fri, 16 Oct 2009 08:10:02 +0000

Acidrip

Today i will be reviewing Acidrip…

Acidrip is a total free and open source software for basic purpose of ripping/transcoding. It is based on its CLI counterpart application name MEncoder, which in itself is quite popular among the CLI users. It is available for both GTK and QTK. It is available for every distro as source code and binary for few like rpm and deb. It is also supported by many package managers but not all as some distros consider it buggy and in an experimental state which has some truth in it.

It is ridiculously simple and easy but it does not offer the level of configuration that Avidemux does as it does not support proprietary codecs and is based only on the ones which are open source like the revered faac and xvid.

It is very easy to learn as it does not have much to learn to do what you want to do and the options are laid out pretty clearly. Although it is very good at many things..it lacks one of the basic things that are required for it to be labelled as user friendly and that thing is presets. That means you will have to fiddle with the options to gain the output you want.

Anyway, general Acidrip window is divided into two parts … one consisting tabs (five of them for different purposes of ripping/transcoding), one of which is always selected and the other consisting statuses.

The following is a general layout of Acidrip:

Status Side of Acidrip:

Status are displayed in right side of Acidrip and do not change even if the tab is changed. I call it the status side although it has no name just the sake for naming it. It actually consists of three parts Video Source, Cache Status, Encoding Status.

Vidoe Source:

It has customizable path in which Acidrip would look for video files. It has a load options to load videos from the path specified. When a source has many files you can choose which one to encode by highlighting that file in the directory content options (the place where the titles are displayed). You should usually choose the file with the highest playing time if there are multiple files which are of the same title but are in different branch of title tree.

Cache Status:

It simply shows cache available and being used by the encoding process

Encoding Status:

It has general encoding status..that is – it has time left, Encoding speed, Estimated resultant file size, Time encoded, Average bitrate and a bar for showing the progress.

Tab Side of Acidrip:

It consists of two parts…one is tab part and the other one I like to main options of part.

Tab Part:

It consist of five tabs namely – General, Video, Preview, Queue and Settings.

They are pretty neatly laid out and are easy to understand even by a novice.

General Tab:

It consists of three parts namely – General, Audio and Other Stuff.

General part of the general tab mainly deals with the file’s physical properties like Track title. Filename. Container(Format), file size, number of files (the number of files to be made out of the Input file) and file info. File Info contains options to edit the name, artist, subject, genre and copyright of the file.

Audio part of the general tab mainly deals with the audio properties of the file like language, codec, gain and options. Language is the option to choose which language the audio the file should be in if there are multiple languages available. Codec lists the various codec which are available to encode the file. Gain is if you find the audio volume level too high or low for your preference and want to change it. Options are the options you want to use for the selected audio codec.

Other stuff part of the general tab deals with the option like subtitles which are available and you want to use. It also contains the options to keep the subtitles in a separate file or to be directly rendered to the video. It also contains Miscellaneous options which are the options which cannot be put anywhere else.

Video Tab:

It contains or four parts – video, crop, scale and filters.

Video part of the video tab mainly consists of everything video like Codec to be used, Passes to be made, Bitrate and Options for configuration.

Crop part of the video tab has the options to crop the video (nothing special here). It also contains an auto-detect option.

Scale part of the video tab contains options to resize the video.

Filter contain options which can be associated with filters. It can use distortion options and the likes

Preview Tab:

It constist of only one part. As the name suggests it is preview. Preview can be important in deciding if you want to remain with the settings you have choosen.

Preview can be showed in separated window or it can be embedded in the Acidrip window. There is also an option to view the preview frame by frame.

Queue Tab:

It consist of uncompleted encoded that are added to queue.

There are options to erase the items in the queue or even export them.

Setting Tab:

It consist of the settings and the options of Acidrip. They do not directly affect the encoding and are irrelevant to our purpose of ripping/encoding so i am not going to include them, but believe me they are very simple and if you want to know about what any options do then you just have to move your mouse over that option and you will be told in text the use of that option. Do try out the settings and customize to your heart’s content as to what should happen before or after encoding. And don’t forget to save them after you have chosen them to you likeness. If you think that you have goofed up or want the original setting than you can simply revert them using the revert button.

Main Options Part:

It contains of four options and they do not change with the change of tab. The options are fairly simple to understand. They are ‘Queue’, ‘Start’, ‘Compact’ and ‘Quit’.

Verdict:

The Guide/Review is before you. The verdict is that Acidrip is very easy to use and has a well laid out layout. But it lack in areas such as presets and configuration that can be achieved. It hardly has any such thing assmall tweaks under its sleeve. It is the best option if you strictly want to stick FLOSS (free/libre open source software which is same as FOSS).

May you find life in yourself…. Ankit Sangwan

Series: Ripping/Transcoding Tools In Linux - Avidemux Video Editor

Ankit Sangwan — Tue, 13 Oct 2009 16:30:20 +0000

Avidemux Video Editor

The first application/tool i would be reviewing is Avidemux Video Editor.

Avidemux is a FOSS program made for multi-purpose video editing and processing. It supports CLI. It is also compatible to both GTK and QTK graphics toolkit. Like all other application which have received good response, it is available in almost all the Distros (and many package managers).

It is a great tool for video authoring and certainly the most powerful in its league. But some say it does not have the ease of use most users would be expecting at the end of the day after you sit down and want a relaxing transcoding of the new dvd you bought which is not exactly true.

Although its is a little inconvenient to learn as there is certainly not a well drafted and publicized manual to all its functions, it has simplicity at its side. Once you have learned its uses, then it is fairly easy to navigate though the maze of options (which are as much as you can wish for but are hidden in nooks and crannies). I say that it is simple because much of the main used option are just a click away(if you manage to find where those clicks should be).

This is a Avidemux window:

It mainly consists 4 areas – Menubar, Main Toolbar, A/V toolbar and video preview area.

Menubar:

Menubar contains 10 menu…namely file, edit, view, video, audio, tools, auto, go, custom, help

File Menu:

File menu contains options like open, append, save, properties, all of which are self explanatory and are concerned about starting projects and saving them.

Edit Menu:

Edit menu contains options to edit general setting of Avidemux, again all the option are self explanatory.

View Menu:

View menu contains option to change the general view of Avidemux, as till now, all the options are self explanatory.

Video Menu:

Video Menu in Avidemux contains video authoring option like decoding, encoding, filters and the likes which are not that self-explanatory to those of you who are new to Ripping/Transcoding field and will be discussed later in this post with some detail (A/V Side Bar Section).

Audio Menu:

Audio Menu in Avidemux contains audio option like track options. It also has encoding and filters options which will be discussed later in this post with some detail (A/V Side Bar Section).

Tools Menu:

It has tools for frame configuration and the like, which i find no need to discuss while giving a short overview. It also has a calculator tool which will be discussed in some detail later (Main Toolbar).

Auto Menu:

Auto menu contains popular presets for video encoding. Presets contain automatic bitrate, crop, filters, codecs to be used, container to be used, etc. It is really useful if you don’t have a clue as to which settings you should choose if you want a certain type of output such as one for IPOD or PSP.

Go Menu:

Go menu in avidemux contains various navigation option in video. It has options to play/stop video , go to a specified frame/time and skip or fast-forward though a video.

Custom Menu:

Custom Menu contain custom user settings…no further explanation avalable.

Help Menu:

It contains knowledge about the installed plugins, version of avidemux, built-in support and the likes.

Main Toolbar:

Main toolbar contains general file options such as open, save and properties as well as tools like calculator. It also contains volume control and file options like input, output, side, separate and the likes.

Calculator Tool:

Calculator tool is a very useful one and probably one of a kind. It is used the calculate the resultant bitrate, size, bits/pixel of a file under certain conditions such as format, duration, audio bitrate, etc.

A/V Side Bar:

This is the main section we are concerned with…A/V Side Bar is the main section in which most of the transcoding can be done. We will be discussing its options in detail.

It consists of three main part…Video, Audio and Format

Video:

It contains three sub parts … video codec, configuration option for the codec and the filters that can be used

Video codec option in avidemux in its A/V side bar contains the various codecs that are supported by Avidemux and can be used. The above lists shows the codec available in avidemux and the copy option is for ripping the media to the hard drive in the format of the original media. Avidemux contains by default the support for all the major codecs including H.264, MP4 for x264 and Xvid…which are pretty popular.

Configuration option for video codecs are different for different codecs but basically it contains bitrate, quality, passes, etc. which are pretty good option to play with if you are an enthusiast but to be left if you don’t know much about video authoring

Filters options contains various filters that can be used. It mainly consists of 8 sections…namely ‘Transform’, ‘Interlacing’, ‘Colors’, ‘Noise’, ‘Sharpness’, ‘Subtitle’, ‘Miscellaneous’ and ‘External’.

Few filter to be taken notice of are ‘Resize’, ‘Crop’ and ‘Subtitles’ as you might end up using them a lot of times as no video is perfect for you preferences.

Audio:

It contains three sub-parts as with Video. They are Codec, Configuration for the codec (which is not different for different codecs as in Video) and Filters that can be used.

Audio codec option in avidemux in it’s A/V side bar contains the various codecs that are supported by Avidemux and can be used. The above lists shows the codec available in avidemux and the copy option is for ripping the audio to the hard drive in the format of the original audio. Audio Codecs in Avidemux are diverse and very useful.

Configure option consists of audio bitrate configuration and that is the beginning and the end of it.

Audio Filter option in A/V side bar Audio section is pretty uncomplicated (simple) and contain various filters like Frame rate change, mixer, gain and the likes.

Frame rate change is simply a frame rate change mode changer. Mixer is for mixing audio and changing its compatibility options ( here I mean the output audio) such as mono, stereo, surround and 5 channel sound. Gain is for gaining or reducing the audio volume. Other options are either self-explanatory or not needed to be tweaked.

Format:

Format in Avidemux A/V Side Bar is for choosing container. Container here meaning the extension it will be using. It is very important as some devices like IPod do not support all formats.

Verdict:

There you have the review and a guide now. The verdict for this one is that it is simple for those who like to just get on with the get over with it and has many tweaking (which i have not explained here at all) for those of you who are enthusiasts. It is overall a good all-rounder.

May you find life in yourself….Ankit Sangwan

Goodbye Mediation Server, Hello NET

Nathan Chapman — Thu, 17 Sep 2009 11:35:04 +0000

Network Equipment Technologies (NET) announced the release of the latest software for the VX Intelligent Voice Gateway this week.

The “Extended UC Features” option availible in this new release means that there is no need to deploy an OCS mediation server with the gateway. All of the functionality that the that the mediation server provided is now built into the VX software (not just a Windows server inside the same chassis), including:

AD lookup for E.164 to SIP URI resolution
RTA/Siren to G.711 transcoding
ICE/STUN/TURN for NAT traversal
SRTP

The “Extended UC Features” option is available from version 4.7.1 (due for release in early October) and consists of a license key and a hardware encryption/accelerator card. The license and card will come pre installed when ordered with a new gateway. Existing VX gateways may be upgraded.

The encryption/accelerator card uses one of the two slots available in the rear of the VX1200 meaning that the E1 capacity is reduced from 8 to 4 interfaces (120 channels). Higher capacity is available in the VX1800 gateway, however I would suggest that organisations that need more than 4 E1 interfaces should consider deploying two VX1200s for high availability.

NET has also implemented some neat features around voice availability and intelligent call routing that leverages its Active Directory integration.

The VX gateways are considered premium gateways both in terms of both features and price. However, the ability to deploy them without having to build and maintain another Windows server platform (which can’t be virtualised) makes them much more affordable.

Now we just need a security hardware vendor to integrate that access edge server role into an appliance

TVersity, my codec your codec my codec

samsungdlna — Mon, 03 Aug 2009 02:15:36 +0000

So I see someone checking for transcoding, I assume with TVersity. Sorry, that takes the pay version of the product which I ain’t gonna do. I am sure others would be interested, so post back here if you want to share your findings.

Remember, graduate soon TVersity user, and graduate often!

Batch Transcoding MP4 Videos to MPEG-2 Format

Adam Craven — Sun, 12 Jul 2009 05:38:59 +0000

I was looking for a way to transcode a bunch of medium quality Mp4 videos into good quality MPEG-2 formatted video so I could then make them into a DVD.

I had problems with my favourite transcoder (MediaCoder) and also a bunch of other transcoders that I unsuccessfully tried.

[Note: I use VLC media player to play my media files. Since VLC has no requirement for codecs to be installed on one’s computer, it turns out I didn’t notice that some of the codecs on my computer were corrupt. After working around the problem with the solution I discuss below, I found that I couldn’t even open the MP4 files in Nero ShowTime – I would get a StackHash_1703 error when it crashed. The root cause of the problem that prevented me from using the transcoders I tried was a corruption of codecs from my installation of Nero. The root cause of the problem has been solved by applying an update to Nero, and now I can once again use my favourite MediaCoder for everything…]

However before I solved that root cause problem, the following is what I did.

I found that AviDemux 1.4 successfully converted the videos to MPEG-2, but version 1.5 would give an error. But, even after using version 1.4 to do the conversion, the MPEG-2 videos were not compliant with DVD standards. From here though I could then use MediaCoder to add the borders I wanted, increase the brightness, increase the volume, deinterlace the video and convert it to 25 frames per second for PAL. But, I wasn’t happy about doing double encoding and so I wanted a better solution.

And so we finally come to the alternative solution that worked very well for me and forced me to learn some lower-level details about transcoding. This solution required the following tools:

ffdshow – a DirectShow and VFW codec for decoding/encoding many video and audio formats
AviSynth 2.5.8 – a powerful tool for video post-production that provides ways of editing and processing videos, and the Decomb plugin
QuEnc 0.72 – a tool that uses a built in FFMPEG library to encode from an AviSynth Script file to a raw MPEG-1/MPEG-2(VCD,SVCD,DVD) video stream

While learning the AviSynth syntax, I found it very useful to open the AviSynth script in VirtualDubMod so I could immediately see and confirm the effects of the script on the video.

I created the following AviSynth user-defined function, and put it into a file named ConvertCommon.avsi:

LoadPlugin(“C:\Program Files (x86)\AviSynth 2.5\plugins\decomb.dll”)

function Convert(string mp4Filename)
{
DirectShowSource(mp4Filename)

BicubicResize(720,408)
ConvertToYV12(interlaced=true)
FieldDeinterlace(show=false) #DECOMB

ConvertFPS(25)

Tweak(bright=10, cont=1.1)

AddBorders(0, 84, 0, 84, $000000)

return AmplifydB(3)
}

This user-defined function is specific to the video processing I wanted to perform on all the MP4 files in my batch. It resizes the video to the size and aspect ratio that I carefully pre-determined, converts the colour to YV12, deinterlaces the video, converts it to 25 frames per second, tweaks the brightness and contrast, adds letterboxing to make the final video size 720×576 for PAL DVD compliance, and amplifies the volume by 3 decibels.

I also stumbled on the following AVS2DVD Batch Script which automates the use of QuEnc on all the AviSynth scripts in a particular folder, and also goes even further to create DVD-Video files. This provided me with a good starting point for automating the conversion of my MP4 files. There were however three issues I has with that script (for my purposes):

I didn’t want to manually create AviSynth Script files for every MP4 file in the batch I wanted to process – because I wanted the exact same post-processing to occur to all the videos.
I needed to edit the resulting MPEG-2 videos so I didn’t want to automatically convert them to DVD-Video files.
It was using the command-line syntax of an older version of QuEnc, which errored in the new version.

Below is the modified batch script which loops through all of the MP4 files in a specified folder, creates a temporary AviSynth script which calls my user-defined function, and passes the temporary script to QuEnc with PAL DVD-compliant settings.

@SET MP4SourceFolder=F:\Videos\Mp4
@SET MPG2OutputFolder=F:\Videos\ConvertedMpeg2

@SET quenc=F:\Videos\QuEnc

@SET QBitrate=3000
@SET QMaxBitrate=9000
@REM Set to the max bitrate minus the sum of all the audio track bitrates. For DVD, 9800-xxx
@SET QDC=10
@SET QPasses=2
@SET QMPEGType=2
@SET QMPEGMuxProfile=DVD
@SET QGOPSize=15
@SET QMaxBFrames=2
@SET QAspectRatio=1:1
@SET QAudioCodec=AC3
@SET QAudioBitrate=192
@REM AC3 @ 192/256 kbps for stereo and 384/448 kbps for 5.1ch is recommended
@SET QHQ=
@REM Leave QHQ blank if you want High Quality turned on, otherwise set it to no
@SET QVBR=
@REM Leave QVBR blank if you want VBR turned on, otherwise set it to no
@SET QSCENE=no
@REM Leave QSCENE blank if you want Scene Detection turned on, otherwise set it to no
@SET QTRELL=
@REM Leave QTRELL blank if you want Trellis Quant turned on, otherwise set it to no
@SET QCGOP=no
@REM Leave QCGOP blank if you want to use Closed GOP, otherwise set it to no
@SET QInterlace=no
@REM Leave QInterlace blank if you want to use Interlaced Encoding, otherwise set it to no
@SET QExtreme=no
@REM Leave QExtreme blank if you want to use Extreme encoding, otherwise set it to no
@SET QCMatrix=no
@SET QCMatrixFile=
@REM If you wish to use a Custom Matrix, leave QCMatrix blank and add your file path to QCMatrixFile,
@REM otherwise set QCMatrix to no, and leave QCMatrixFile blank.

@SET QCMD=-auto -close
@REM Use for additional parameters such as Silent, Auto, Close and Shutdown

FOR /F “usebackq delims==” %%I IN (`dir /b “%MP4SourceFolder%\*.mp4″`) DO (
@REM Make AVS file
@echo Import^(“ConvertCommon.avsi”^) 1> temp.avs
@echo Convert^(“%MP4SourceFolder%\%%~nxI”^) 1>> temp.avs

@REM Compress AVS file to MPEG2 using QuENC
“%quenc%\quenc.exe” -i “.\temp.avs” -o “%MPG2OutputFolder%\%%~nI.mpg” -b %QBitrate% -maxbitrate %QMaxBitrate% -dc %QDC% -%QPasses% -%QHQ%hq -%QVBR%vbr -%QSCENE%scene -%QTRELL%trell -%QCGOP%cgop -%QInterlace%interlaced -%QExtreme%extreme -gopsize %QGOPSize% -maxbframes %QMaxBFrames% -%QCMatrix%cmatrix %QCMatrixFile% -aspectratio %QAspectRatio% -mpeg%QMPEGType%mux %QMPEGMuxProfile% -audiocodec %QAudioCodec% -audiobitrate %QAudioBitrate% %QCMD%
)

pause

del temp.avs

And after running the batch script over a few days, my transcoding is now complete!

Ubuntu Forums - DVD AVI DivX/Xvid

bigweeds — Fri, 12 Jun 2009 16:13:04 +0000

Howto: set-up your own DVD transcoding cluster

http://georgia.ubuntuforums.org/showpost.php?p=6057925&postcount=16

I have a 500GB XBOX, which is connected to a HDTV with composite video and optical 5.1 sound which I use to watch DVD’s, using XBMC. It is really nice system, and the only problem I have is the time it takes to transcode DVD to AVI for play on the XBOX (that, and the time it takes to copy the AVI files over FTP to the XBOX)

Since I had a few AMD64 X2 Dual Core motherboards and CPU’s lying around, I was looking at ways to user them in a cluster/networked environment when I came across the DVD::RIP program, and loved the idea that you could transcode with a cluster, reducing the processing time significantly.

However, once I started down the process of configuring my cluster, it was to time consuming and complicated (for me in any case), and as I wanted something where I could just join a PC (node) to the cluster through PXE without affecting the current operating system on that PC (if any) and/or use diskless, headless PC’s as a node in a DVD::RIP cluster.

So, after many false starts I came upon DRBL through a Google search for PXE.

Starting Point:

I have decided to have a single master for DVD::RIP and DVD::RIP Clone Master, and the DRBL server, with separate nodes.

Please note this installation is for the i386 platform, and may not work for other platforms.

Master Server:

DRBL requires your DRBL server to have two network cards, one for the internet (you do need his if you intend to download the cluster boot images) and one for the cluster. If you do not have two network cards, this method will not work, but you can attempt two network addresses per Ethernet card (eth0 and eth0:1), but I cannot help you there

So, you have your two network cards in the machine, pop the Ubuntu CD into your CD/DVD drive and install Ubuntu 8.04 Hardy Heron as normal (if you have not done so already)

Once your machine is up and running and all updates installed, check that the Universe and Multiverse repositories are enabled (in my case it was by default, and you should not have any problems).

If you want to, you can check this by either looking at the /etc/apt/sources.list, if you are that way inclined, or you can use the Synaptics Package Manager menus. Install the following software, the sequence is not too important:

Code:

  sudo apt-get install dvdrip
  sudo apt-get install gxine
  sudo apt-get install mplayer
  sudo apt-get install rar
  sudo apt-get install ogle
  sudo apt-get install vlc

Download libdvdcss.deb (at your own risk) from http://www.dtek.chalmers.se/groups/d…2.5-1_i386.deb and install using debpackager (double click on the downloaded file, and it should ask you to open it with debpackager as the default, unless you have tinkered with your settings)

The next section deals with the workhorse behind the whole process, namely DRBL, which stands for Diskless Remote Boot Linux

Installing DRBL:

The official DRBL installation how-to can be found at http://drbl.sourceforge.net/one4all/

I will reproduce the steps I needed to take to get it to work. It is surprisingly easy, considering the complexities, and I must congratulate the authors on a very useful piece of software. In any case, here goes:

To allow access to their repository you have to import the key. First download the key as follow:

Code:

wget http://drbl.nchc.org.tw/GPG-KEY-DRBL

Thereafter, import the key with:

Code:

apt-key add GPG-KEY-DRBL

You will also have to update your repository source list. Open a terminal window, and type:

Code:

sudo gedit /etc/apt/sources.list

Once the file is displayed, scroll down to the bottom of the file, and place your cursor in a new line. Copy the two lines of text below (select/highlight the text, right click on it, and select copy) and paste it into the already open sources.list file by right clicking at the cursor and selecting paste. Once you have done that, save the file (File->Save) and close gedit.

I suggest you configure your two network cards at this point in time. The one card will be used for internet access, and the other for cluster traffic. I recommend you get yourself Gigabit Ethernet cards and hubs to benefit from the speed increase. I have found my Gigabit network to be my main bottleneck on a 1 Master + 6 Node Cluster, with the network often maxing out, slowing data transfer and hence transcoding.

You can configure your network in two ways, through the graphical network manager in Ubuntu, or by configuring the network interfaces by editing the /etc/network/interfaces file.

I chose to use the graphical network manager after struggling for hours to configure the network in the more traditional manner.

You have to select a different private network for your cluster network on eth1. My ADSL router supplies DHCP to my home network in the 192.168.1.xxx range, and I set my eth1 to the following:

Address: 192.168.100.1
Netmask: 255.255.255.0

To use the graphical network manager, left click the network panel indicator (normally on the top panel at the right hand side). Click on the UNLOCK button and unlock the panel with your password.

You should see Wired Connection eth0 and Wired Connection eth1 in your list of network interfaces. You may have others (such as ath0 etc, but I cannot tell how well those interfaces will work)

Select eth1 and click on PROPERTIES. Disable roaming mode and select Static IP address under the CONFIGURATION dropdown. In IP address type 192.168.100.1(or any other private IP address you wish, as long as it is not the same network range as eth0).

If you are not sure of your network address for eth0, just open up a terminal window and type ifconfig and press enter, and you will be able to see the network configuration of your computer.

To complete the configuration on eth1, under subnet mask enter 255.255.255.0. (I am sure there are networking geniuses who could perhaps correct my choices (there may be reasons to be on another subnet mask etc that could be beneficial for performance and any input would be appreciated).

Do not worry about gateway or any other information, just click OK, and close the application.

Once you have done that, you could restart your network by opening your terminal window and type:

Code:

sudo /etc/networking restart

After this is completed, check your network configuration by typing ifconfig and ensure your two network cards are configured correctly.

Installing the DRBL server

Now it is time to download and install DRBL server. This is simply done by opening your terminal window (you can use Synaptics if you prefer) and typing:

Code:

sudo apt-get install drbl

Configuring the DRBL server – PART 1

There are two ways of doing it (at least). One method is a step by step approach where you can configure all the settings yourself, and a second option that will configure the system to a default configuration. I will explain option 1 in more detail, and will only give a brief outline of option 2.

Option 2 – For the impatient

In your terminal window, type:

Code:

sudo /opt/drbl/sbin/drbl4imp

and go and have a cup of coffee (or two, depending on your download speed) whilst the boot images are downloaded, and the system configured. I have not tried this option yet, and your feedback would be appreciated if you decide to take this route.

This configuration route will assume 12 cluster nodes/clients per network card (excluding eth0), and will accept all the default options (which may or may not work) for a DVD::RIP cluster. If it completes successfully, you may, or may not, have a working cluster at the end.

Option 1 – My preferred route:

Once you have installed the DRBL server, open your terminal window and enter:

Code:

/opt/drbl/sbin/drblsrv –i

This will start an interactive session, requiring your input. My configuration was done as follow:

Hint! When a yes/no option is available, the default value is uppercase, Ex. (y/N), the default is “N”, when you press “Enter”, it will use “N”. If you are not sure which one to choose, you can just press “Enter” key.

************************************************** ***.

Installing DRBL for Debian Linux…

************************************************** ***.

Do you want to install those network installation boot images so that you can let client to install some GNU/Linux distributions (Debian, Ubuntu, RedHat Linux, Fedora Core, Mandriva, CentOS and OpenSuSE…) via network ? ///NOTE/// This action will download a lot of files (> 100 MB totally) from Internet, so it might take a few minutes. If your client machine has harddisk and it is possible you will install GNU/Linux into that, say Y here. If you say “no” here, feel free to run drbl-netinstall to install them later.

[y/N] Y

************************************************** ***.

This GNU/Linux distribution uses one kernel to support SMP and non-SMP arch.

************************************************** ***.

Do you want to use the serial console output for clients ?

If you do NOT know anything about this, say “N” here, otherwise clients might show NOTHING on the screen !

[y/N] N

************************************************** ***.

Which CPU architecture kernel do you want to assign for DRBL clients ?

0 -> i386 level CPU

1 -> i586 level CPU

2 -> Use the same CPU level with that of this DRBL server

Note! Note Note! Note! Note! Note! Note!

NOTE!!! If your client machine(s) is not the same level with server, please answer “0″ or “1″, otherwise your client machine(s) will NOT be able to boot.

If you use wrong CPU level kernel, the glibc and openssl package might use i686 or i386, the kernel might use i686, i586 or i386, which might be not suitable to all your machines.

If you are not sure, “1″ is recommended, this will still have good performance and compatibility.

[1] 2

(THIS IS IMPORTANT AS A BUG IN THE UBUNTU PACKAGE LIST CONTAINS A REFERENCE TO A MISSING LINUX-IMAGE-2.6.24-20-386, WHICH WILL CAUSE THE PROGRAM TO ABORT. I SPENT 8 HOURS, THREE COMPLETE REINSTALLS AND MANY MANY SERVER INSTALLATION ATTEMPTS TO FIND OUT WHY!!!)

************************************************** ***.

The CPU arch you specify: 0

No optimization for your system, we will use the “i386″ package.

************************************************** ***.

Cleaning the cache of apt to make some settings effect…

Hit http://security.ubuntu.com hardy-security Release.gpg

Hit http://gb.archive.ubuntu.com hardy Release.gpg

Hit http://gb.archive.ubuntu.com hardy-updates Release.gpg

Hit http://security.ubuntu.com hardy-security Release

Hit http://gb.archive.ubuntu.com hardy Release

Hit http://gb.archive.ubuntu.com hardy-updates Release

Hit http://security.ubuntu.com hardy-security/main Packages

Hit http://gb.archive.ubuntu.com hardy/main Packages

Hit http://security.ubuntu.com hardy-security/restricted Packages

Hit http://security.ubuntu.com hardy-security/main Sources

Hit http://gb.archive.ubuntu.com hardy/restricted Packages

Hit http://gb.archive.ubuntu.com hardy/main Sources

Hit http://gb.archive.ubuntu.com hardy/restricted Sources

Hit http://security.ubuntu.com hardy-security/restricted Sources

Hit http://security.ubuntu.com hardy-security/universe Packages

Hit http://security.ubuntu.com hardy-security/universe Sources

Hit http://security.ubuntu.com hardy-security/multiverse Packages

Hit http://security.ubuntu.com hardy-security/multiverse Sources

Hit http://gb.archive.ubuntu.com hardy/universe Packages

Hit http://gb.archive.ubuntu.com hardy/universe Sources

Hit http://gb.archive.ubuntu.com hardy/multiverse Packages

Hit http://gb.archive.ubuntu.com hardy/multiverse Sources

Hit http://gb.archive.ubuntu.com hardy-updates/main Packages

Hit http://gb.archive.ubuntu.com hardy-updates/restricted Packages

Hit http://gb.archive.ubuntu.com hardy-updates/main Sources

Hit http://gb.archive.ubuntu.com hardy-updates/restricted Sources

Hit http://gb.archive.ubuntu.com hardy-updates/universe Packages

Hit http://gb.archive.ubuntu.com hardy-updates/universe Sources

Hit http://gb.archive.ubuntu.com hardy-updates/multiverse Packages

Hit http://gb.archive.ubuntu.com hardy-updates/multiverse Sources

Hit http://free.nchc.org.tw hardy Release.gpg

Hit http://free.nchc.org.tw drbl Release.gpg

Hit http://free.nchc.org.tw hardy Release

Hit http://free.nchc.org.tw drbl Release

Hit http://free.nchc.org.tw hardy/main Packages

Hit http://free.nchc.org.tw hardy/restricted Packages

Hit http://free.nchc.org.tw hardy/universe Packages

Hit http://free.nchc.org.tw hardy/multiverse Packages

Hit http://free.nchc.org.tw drbl/stable Packages

Reading package lists… Done

************************************************** ***.

Do you want to upgrade operating system ?

[y/N]
N

************************************************** ***.

2nd, installing the necessary files for DRBL…

************************************************** ***.

Searching if lvm2 ntfs-3g lshw available…

Package lvm2 exists in repository.

Package ntfs-3g exists in repository.

Package lshw exists in repository.

Reading package lists… Done

Building dependency tree

Reading state information… Done

util-linux is already the newest version.

tar is already the newest version.

gzip is already the newest version.

bzip2 is already the newest version.

procps is already the newest version.

dialog is already the newest version.

rsync is already the newest version.

parted is already the newest version.

pciutils is already the newest version.

tcpdump is already the newest version.

bc is already the newest version.

grub is already the newest version.

gawk is already the newest version.

hdparm is already the newest version.

sdparm is already the newest version.

netcat is already the newest version.

file is already the newest version.

ethtool is already the newest version.

etherwake is already the newest version.

ssh is already the newest version.

syslinux is already the newest version.

mtools is already the newest version.

Note, selecting genisoimage instead of mkisofs

genisoimage is already the newest version.

reiserfsprogs is already the newest version.

e2fsprogs is already the newest version.

psmisc is already the newest version.

locales is already the newest version.

wget is already the newest version.

disktype is already the newest version.

zip is already the newest version.

unzip is already the newest version.

patch is already the newest version.

initscripts is already the newest version.

dhcp3-server is already the newest version.

tftpd-hpa is already the newest version.

nfs-kernel-server is already the newest version.

nis is already the newest version.

curl is already the newest version.

lftp is already the newest version.

iptables is already the newest version.

libdigest-sha1-perl is already the newest version.

lvm2 is already the newest version.

ntfs-3g is already the newest version.

lshw is already the newest version.

partclone is already the newest version.

mkpxeinitrd-net is already the newest version.

clonezilla is already the newest version.

mkswap-uuid is already the newest version.

drbl-partimage is already the newest version.

drbl-ntfsprogs is already the newest version.

drbl-chntpw is already the newest version.

drbl-lzop is already the newest version.

udpcast is already the newest version.

drbl-etherboot is already the newest version.

freedos is already the newest version.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

************************************************** ***.

Trying to upgrade some necessary packages if available…

************************************************** ***.

In ayo repository, searching the latest kernel …

The kernel image in Ubuntu 8.04 “uses generic” for i686/amd64 CPU.

The latest kernel in the ayo repository is linux-image-2.6.24-20-generic

There are 2 kernels available for clients, which one do you prefer ?

[1]: kernel 2.6.24-19-generic i586 (from this DRBL server)

[2]: linux-image-2.6.24-20-generic (from apt repository)

[1]
1 (SELECT 1 HERE FOR THE REASONS GIVEN ABOVE)

Clients will use the kernel 2.6.24-19-generic i586 from server.

It might take several minutes to install this kernel, please be patient…

done!

************************************************** ***.

Install kernel for clients… …

In ayo repository, searching the latest kernel …

************************************************** ***.

Now run: drblsrv-offline -c -d -a -l en_US -s 2.6.24-19-generic “” “”

Using kernel from this server for client…

************************************************** ***.

The version number for your OS: Ubuntu 8.04

************************************************** ***.

Install kernel for clients… …

The kernel for client is copied from server.

Installing kernel 2.6.24-19-generic for clients…

It might take several minutes to install this kernel, please be patient… …done!

Generating modules.dep and map files for clients… done!

************************************************** ***.

Creating config file for PXE clients…

Copying pxelinux.0, menu.c32, vesamenu.c32, chain.c32 and memdisk to /tftpboot/nbi_img…

Copying memtest86+ to /tftpboot/nbi_img…

Copying FreeDOS files to /tftpboot/nbi_img/…

Generating default pxelinux config (/tftpboot/nbi_img/pxelinux.cfg/default)…

Use com32 module: vesamenu.c32

Adding menus for DRBL, local boot, memtest86+, FreeDOS…

Adding netinstall-CentOS-4.6-i386 menu…

Adding netinstall-CentOS-4.6-x86_64 menu…

Adding netinstall-Debian-etch-amd64 menu…

Adding netinstall-Debian-etch-i386 menu…

Adding netinstall-Debian-lenny-amd64 menu…

Adding netinstall-Debian-lenny-i386 menu…

Adding netinstall-Fedora-9-i386 menu…

Adding netinstall-Fedora-9-x86_64 menu…

Adding netinstall-Mandriva-2008.1-i586 menu…

Adding netinstall-Scientific-5.1-i386 menu…

Adding netinstall-Scientific-5.1-x86_64 menu…

Adding netinstall-Ubuntu-gutsy-amd64 menu…

Adding netinstall-Ubuntu-gutsy-i386 menu…

Adding netinstall-Ubuntu-hardy-amd64 menu…

Adding netinstall-Ubuntu-hardy-i386 menu…

Adding netinstall-openSUSE-10.3-i386 menu…

Adding netinstall-openSUSE-10.3-x86_64 menu…

done!

************************************************** ***.

Creating the image files for PXE and Etherboot client, this will take a few minutes …

The latest kernel for DRBL clients is 2.6.24-19-generic

Running mknic-nbi –kernel 2.6.24-19-generic –all –no-modules

Will client check DHCP server name is “drbl” or not: yes

The maximum times to try to get IP address for a client: 3

The pause time after network card is up: 0

Setting port for udhcpc request to default…

Using the kernel modules from /tftpboot/node_root//lib/modules…

The selected kernel for DRBL clients is: 2.6.24-19-generic

Kernel 2.6 is found, so default to use initramfs.

Creating the network boot initrd for PXE clients by: mkpxeinitrd-net -k 2.6.24-19-generic -t initramfs

Use kernel modules from /tftpboot/node_root//lib/modules/2.6.24-19-generic.

Creating the initRAMFS image…

Initramfs, remove ramdisk_size/ramdisk_block in /tftpboot/nbi_img/pxelinux.cfg/default if exists…

Finished!

Done!

************************************************** ***.

Done!

And that is part 1 done!!!

Configuring the DRBL server – PART 2

The second part is required to create the file system used by the DVD::RIP cluster nodes. First some homework is required to allow connection between the cluster server and the nodes. To achieve this we need to configure ssh security between the server and clients.

Open your terminal window again, and type:

Code:

ssh-keygen

Make sure you enter nothing, nada, niks, buggerol in the passphrase

cluster@ClusterMaster:~$ ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/home/yourusername/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/yourusername/.ssh/id_rsa.

Your public key has been saved in /home/yourusername/.ssh/id_rsa.pub.

The key fingerprint is:

54:8d:41:33:7a:10:bc:4b:5b:60:25:53:c6:5d:16:96 cluster@ClusterMaster

To ensure the system can connect with a password, we have to copy the newly created id_rsa.pub file to a new file called authorized_keys in the /home/yourusername/.ssh directory (replace cluster with your username in this case)

Open your terminal window and type:

Code:

sudo cp /home/yourusername/.ssh/id_rsa.pub /home/yourusername/.ssh/authorized_keys

That is it. As all nodes will use the same home directory, they will automagically pick up the authorized keys from this file, and allow seamless connection to the server.

To make sure the NFS folder is available to the cluster nodes, I recommend you make the following modifications to your drbl configuration file:

In your terminal window, type:

Code:

sudo gedit /opt/drbl/conf/drbl.conf

Near the end of the file, look for a line that contains

Code:

diskless_root_dir_rw_user_add=""

modify this line to read

Code:

diskless_root_dir_rw_user_add="/home/yourusername/dvdrip-data/"

save the file and exit gedit.

Now, to do the final configuration, in a terminal window type:

Code:

sudo /opt/drbl/sbin/drblpush –i

and follow the prompts:

************************************************** ****

Searching the installed packages for DRBL server…This might take several minutes…

Finished searching the installed packages for DRBL server.

************************************************** ****

——————————————————

The interactive mode let you supply the information of your DRBL environment.

——————————————————

Please enter DNS domain (such as drbl.sf.net):

[drbl.sf.net] cluster.net (THIS IS NOT TOO IMPORTANT WHAT YOU CALL IT)

Set DOMAIN as cluster.net

——————————————————

Please enter NIS/YP domain name:

[dvdcluster]
Use the default entry here

Set DOMAIN as dvdcluster

——————————————————

Please enter the client hostname prefix:

This prefix is used to automatically create hostname for clients. If you want to overwrite some or all automatically created hostnames, press Ctrl-C to quit this program now, edit /opt/drbl/conf/client-ip-hostname, then run this program again.

[dvdmaster]
ClusterNode (You can choose anything you want)

Set the client hostname prefix as ClusterNode
——————————————————

Found private IP “192.168.1.11″ in eth0 on your system!

Found private IP “192.168.100.10″ in eth1 on your system!

Configured ethernet card(s) found in your system: eth0 eth1

——————————————————

The public IP address of this server is NOT found.

Which ethernet port in this server is for public Internet access, not for DRBL connection ?

Available ethernet ports in this server:

eth0 (192.168.1.11), eth1 (192.168.100.10),

[eth0]
eth0 (IMPORTANT)

The ethernet port you choose for the WAN connection: eth0

The ethernet port(s) for DRBL environment: eth1

************************************************** ****

Now we can collect the MAC address of clients!

If you want to let the DHCP service in DRBL server offer same IP address to client every time when client boot, and you never did this procedure, you should do it now!

If you already have those MAC addresses of clients, you can put them into different group files (These files number is the same number of networks cards for DRBL service). In this case, you can skip this step.

This step helps you to record the MAC addresses of clients, then divide them into different groups. It will save your time and reduce the typos.

The MAC addresses will be recorded turn by turn according to the boot of clients,

and they will be put into different files according to the network card in server, file name will be like macadr-eth1.txt, macadr-eth2.txt… You can find them in directory /etc/drbl.

Please boot the clients by order, make sure they boot from etherboot or PXE!

Do you want to collect them ?

[y/N]
N

************************************************** ****

OK! Let’s continue…

************************************************** ****

Do you want to let the DHCP service in DRBL server offer same IP address to the client every time when client boots (If you want this function, you have to collect the MAC addresses of clients, and save them in file(s) (as in the previous procedure)). This is for the clients connected to DRBL server’s ethernet network interface eth1 ?

[y/N]
N

************************************************** ****

OK! Let’s continue, we will set the IP address of clients by “first boot gets IP first” instead of fixed one!

************************************************** ****

What is the initial number do you want to use in the last set of digits in the IP (i.e. the initial value of d in the IP address a.b.c.d) for DRBL clients connected to this ethernet port eth1.

[1]
10
(You could use anything other than your initial ip address you set earlier, 10 is safe)

************************************************** ****

How many DRBL clients (PC for students) connected to DRBL server’s ethernet network interface eth1 ?

Please enter the number:

[12] 12

************************************************** ****

The final number in the last set of digits in the clients’ IP is “21″.

We will set the IP address for the clients connected to DRBL server’s ethernet network interface eth1 as: 192.168.100.10 – 192.168.100.21

Accept ? [Y/n]
Y

************************************************** ****

OK! Let’s continue…

************************************************** ****

The Layout for your DRBL environment:

************************************************** ****

NIC NIC IP Clients

+———————————-+

| DRBL SERVER |

| |

| +– [eth0] 192.168.1.11 +- to WAN

| |

| +– [eth1] 192.168.100.10 +- to clients group 1 [ 12 clients, their IP

| | from 192.168.100.10 - 192.168.100.21]

+———————————-+

************************************************** ****

Total clients: 12

************************************************** ****

Press Enter to continue…

——————————————————

In the system, there are 3 modes for diskless linux services:

[0] Full DRBL mode, every client has its own NFS based /etc and /var.

[1] DRBL SSI (Single system image) mode, every client uses tmpfs based /etc and /var. In this mode, the loading and necessary disk space of server will be lighter. NOTE! (a) The client machine memory is recommended at least 256 MB. (b) The setting and config files of client will not be saved to the DRBL server! They are just used once and will vanish after the machine shutdowns! Besides, if you modify any file in the template client (located in /tftpboot/nodes), you have to run /opt/drbl/sbin/drbl-gen-ssi-files to create the template tarball in /tftpboot/node_root/drbl_ssi/. (c) If you want to provide some file to overwrite the setting in the template tarball when client boots, check /tftpboot/node_root/drbl_ssi/clients/00_README for more details.

[2] I do NOT want to provide diskless Linux service to client.

Which mode do you prefer ?

[0]
1
(THIS IS THE PREFERRED ROUTE IF YOU DO NOT WANT TO TOUCH THE INSTALLED OPERATING SYSTEM OF YOUR NODE, OR IF YOU USE CHEAP, DISKLESS NODES)

DRBL SSI mode is set, an elegant mode for DRBL is on the way!

************************************************** ****

——————————————————

In the system, there are 3 modes available for clonezilla:

[0] Full Clonezilla mode, every client has its own NFS based /etc and /var.

[1] Clonezilla box mode, every client uses tmpfs based /etc and /var. In this mode, the loading and necessary disk space of server will be lighter than that in Full Clonezilla mode. Note! In Clonezilla box mode, the setting and config files of client will not be saved to the DRBL server! They just use once and will vanish after the machine shutdowns!

[2] I do NOT want clonezilla.

Which mode do you prefer ?

[0] 2

No Clonezilla is the system!

************************************************** ****

——————————————————

If there is a local harddrive with swap partition or writable file system in your client machine,

do you want to use that swap partition or create a swap file in the writable filesystem so that client has more memory to use ? (This step will NOT destroy any data in that harddisk)

[Y/n]
N

No Clonezilla is the system!

************************************************** ****

——————————————————

If there is a local harddrive with swap partition or writable file system in your client machine,

do you want to use that swap partition or create a swap file in the writable filesystem so that client has more memory to use ? (This step will NOT destroy any data in that harddisk)

[Y/n] N

************************************************** ****

——————————————————

Which mode do you want the clients to use after they boot ?

“1″: Graphic mode (X window system) (default),

“2″: Text mode.

[1] 2

The clients will use text mode when they boot.

************************************************** ****

——————————————————

Do you want to set the root’s password for clients instead of using same root’s password copied from server ? (For better security)

[y/N] N

OK! Let’s continue…

——————————————————

Do you want to set the pxelinux password for clients so that when client boots, a password must be entered to startup (For better security)

[y/N] N

OK! Let’s continue…

——————————————————

Do you want to set the boot prompt for clients ?

[Y/n] Y

How many 1/10 sec is the boot prompt timeout for clients ?

[70] 10

OK! Let’s continue…

——————————————————

Do you want to use graphic background for PXE menu when client boots ?

Note! If you use graphical PXELinux menu, however client fails to boot, you can switch to text mode by running “/opt/drbl/sbin/switch-pxe-bg-mode -m text”.

[y/N] N

Use graphic PXE Linux menu for client.

——————————————————

Do you want to let audio, cdrom, floppy, video and plugdev (like USB device) open to all users in the DRBL client ? If yes, we will add all the users to those device groups in the server and client.

[Y/n] N

——————————————————

By using alias interface, every client can have 2 IPs,

one of them is private IP for clients connected to DRBL server, and the other is public IP for clients directly connected to WAN from switch!

Do you want to setup public IP for clients ?

[y/N] N

——————————————————

Do you want to let DRBL clients have an option to run terminal mode ? i.e. you want to let that client run remote display (which will mostly use resources of server), say “Y” here.

Note!

0. If you say yes to this option, this will be a very limited environment for client, i.e. NO local access for USB, CD, audio, printer, etc. in client.

1. If your server is not powerful, say “no” here.

2. By saying “yes” here, we will turn on xdmcp,

It is never a safe thing to turn on that. Setting up /etc/hosts.allow and /etc/hosts.deny to only allow local access is another alternative but not the safest.

Firewalling port 177 is the safest if you wish to have xdmcp on.

Read the manual for more notes on the security of XDMCP.

Please set it by yourself!

3. If you say “yes” here, you might have to restart your desktop environment manager (gdm/kdm) later, remember to save your data before you close applications!

Do you want to let client has an option to run terminal mode ?

[y/N] N

OK! Let’s continue…

——————————————————

Do you want to let DRBL server as a NAT server ? If not, your DRBL client will NOT be able to access Internet.

[Y/n] N

This DRBL server does NOT provide NAT service, so your DRBL client will NOT be able to access Internet.

************************************************** ****

The running kernel in the server supports NFS over TCP!

Note! If you change the running kernel in the server, and not sure whether the kernel supports NFS over udp or tcp, you’d better to re-run “drblpush -i” again to avoid the client boots in failure!

Press Enter to continue...

——————————————————

Searching installed Etherboot files for dhcpd.conf… done!

************************************************** ****

The calculated NETWORK for eth1 is 192.168.100.0.

* The client IP address you assigned is same with nfsserver: 192.168.100.10,

—> the next IP address (192.168.100.11) will be used for that client!

************************************************** ****

We are now ready to deploy the files to system!

Do you want to continue ?

Warning! If you go on, your firewall rules will be overwritten during the setup!

The original rules will be backed up as iptables.drblsave in system config directory (/etc/sysconfig or /etc/default).

[Y/n]
Y

——————————————————

Checking the necessary disk space… done!

Copying the config file to /etc/drbl… done!

Backup the original /etc/hosts as /etc/hosts.drblsave… done!

Generate the /etc/hosts for clients connected to eth1… done!

Cleaning the stale files of the diskless nodes if they exist… done!

************************************************** ***.

The version number for your GNU/Linux: DBN-TU

Completely cleaning old common root files if they exist… done !

Completely cleaning old nodes if they exist… done !

Creating common root files… This might take several minutes……….. done!

Update the kernel for client if necessary…

The DRBL client uses i586 kernel with version 2.6.24-19-generic…

Trying to update the /tftpboot/node_root/lib/modules/2.6.24-19-generic from server’s /lib/modules/… This might take several minutes…

Found kernel modules in /lib/modules/2.6.24-19-generic and its arch “i586″ matches client’s “i586″…

Syncing /lib/modules/2.6.24-19-generic to client’s common root…

Syncing /boot/*-2.6.24-19-generic* to client’s common root…

Generating the /tftpboot/node_root/lib/modules/2.6.24-19-generic/modules.dep

Copying the directory /etc/ to clients common root /tftpboot/node_root…

Cleaning the ssh key file ssh_host_dsa_key copied from server… done!

Cleaning the ssh key file ssh_host_dsa_key.pub copied from server… done!

Cleaning the ssh key file ssh_host_rsa_key copied from server… done!

Cleaning the ssh key file ssh_host_rsa_key.pub copied from server… done!

Commenting the TCPwrapper related file /tftpboot/node_root/etc/hosts.deny copied from server… done!

Commenting the TCPwrapper related file /tftpboot/node_root/etc/hosts.allow copied from server… done!

The startup services for DRBL client are:

firstboot portmap nis nfs-common ssh hal dbus acpid acpi-support usplash cupsys drblthincli arm-wol sendsigs umountfs

Using udev for clients… Set text mode for Debian DRBL client…

Deleting the accounts (except root) in the clients common root template… done!

Enabling the NIS client in the common root template… done!

Creating some necessary files in the clients common root template……. done!

Creating DRBL client: ClusterNode111 192.168.100.11… Generating SSH host keys for client 192.168.100.11 if they do not exist… done!

Display manager:”gdm”…

Setting node 192.168.100.11 as normal_login… done!

Creating DRBL client: ClusterNode112 192.168.100.12… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode113 192.168.100.13… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode114 192.168.100.14… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode115 192.168.100.15… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode116 192.168.100.16… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode117 192.168.100.17… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode118 192.168.100.18… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode119 192.168.100.19… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode120 192.168.100.20… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode121 192.168.100.21… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Creating DRBL client: ClusterNode122 192.168.100.22… Pseudo client is created for DRBL SSI or clonezilla box mode! done!

Template client for DRBL SSI is 192.168.100.11

Using template host /tftpboot/nodes/192.168.100.11

Generating SSH host keys for client 192.168.100.11 if they do not exist… done!

Generating the files for DRBL single system image template… etc… var… opt/drbl… Root’s openssh public key… done!

Disable the password in pxelinux simple menu for all clients…

Disabling PXE password in config file /tftpboot/nbi_img/pxelinux.cfg/default…

done!

Now add necessary services to this DRBL server: DHCP, TFTP, NFS, NIS…

Generating the NFS exports for DRBL clients…

Backup the original /etc/exports as /etc/exports.drblsave

Exporting to clients by IP address line-by-line…

The /etc/exports setting is ok now!

This DRBL server does NOT provide NAT service, so your DRBL client will NOT be able to access Internet.

Now stop the NAT service…

Flushing firewall rules: success

Now set the YP securenets…

Backup the original /etc/ypserv.securenets as /etc/ypserv.securenets.drblsave

The /etc/ypserv.securenets setting is done!

Update YP…

Now add the service: portmap dhcp3-server nis nfs-common nfs-kernel-server tftpd-hpa drbl-clients-nat

Force to add portmap service in this Debian DRBL server…

Force to add dhcp3-server service in this Debian DRBL server…

Force to add nis service in this Debian DRBL server…

Force to add nfs-common service in this Debian DRBL server…

Force to add nfs-kernel-server service in this Debian DRBL server…

Force to add tftpd-hpa service in this Debian DRBL server…

Force to add drbl-clients-nat service in this Debian DRBL server…

Now start the service: portmap dhcp3-server nis nfs-common nfs-kernel-server tftpd-hpa drbl-clients-nat

* Stopping portmap daemon… [ OK ]

* Starting portmap daemon… [ OK ]

* Stopping DHCP server dhcpd3 [ OK ]

* Starting DHCP server dhcpd3 [ OK ]

* Starting NIS services [ OK ]

* Stopping NFS common utilities [ OK ]

* Starting NFS common utilities [ OK ]

* Stopping NFS kernel daemon [ OK ]

* Unexporting directories for NFS kernel daemon… [ OK ]

* Exporting directories for NFS kernel daemon… [ OK ]

* Starting NFS kernel daemon [ OK ]

Restarting HPA’s tftpd: in.tftpd.

Stopping the NAT services for DRBL clients… Now stop the NAT service…

Flushing firewall rules: success

done!

Starting the NAT services for DRBL clients… done!

Turn on ip_forward now.

The display manager in this DRBL server is “gdm”

The GDM remote access in the DRBL server is already off!

Disable the terminal mode for DRBL clients …

done !

Clean all the previous saved config file if they exist…done!

Turn on the boot prompt for PXE client…done!

Turn off the thin client option in PXE boot menu…done!

Modifying /tftpboot/nbi_img/pxelinux.cfg/default to let DRBL client use text PXE boot menu… done!

DRBL SSI mode. Set clientdir opt for label drbl in pxelinux config…

Setting drbl_mode=”drbl_ssi_mode” in /etc/drbl/drbl_deploy.conf and /etc/drbl/drblpush.conf… done!

Clonezilla service is set as unavailable. Set clientdir opt for label clonezilla in pxelinux config…

Setting clonezilla_mode=”none” in /etc/drbl/drbl_deploy.conf and /etc/drbl/drblpush.conf… done!

************************************************** ***.

Enjoy DRBL!!!

http://drbl.nchc.org.tw; http://drbl.sf.net

NCHC Free Software Labs, Taiwan. http://free.nchc.org.tw

************************************************** ***.

If you like, you can reboot the DRBL server now to make sure everything is ready…(This is not necessary, just an option.).

************************************************** ***.

DRBL server is ready! Now set the client machines to boot from PXE or Etherboot (refer to http://drbl.sourceforge.net for more details).

NOTE! If Etherboot is used in client machine, version 5.4.0 or newer is required!

PS. The config file is saved as /etc/drbl/drblpush.conf. Therefore if you want to run drblpush with the same config again, you may run it as: /opt/drbl/sbin/drblpush -c /etc/drbl/drblpush.conf

THAT IS IT, THE DRBL SERVER IS ACTIVE AND WAITING FOR CLIENTS!!!
You can at anytime rerun both Part 1 and Part 2 to change or fix your configuration.

Configuring the NODES

To use the server, you will have to configure your nodes to boot with PXE (over ethernet). This is set in your computer’s BIOS, which is accessed normally by pushing DEL or F2 or whatever weird keystroke combination the manufacturers of your motherboard chose to use. Please see your manual for your motherboard or a friend to help you in this regard.

Once they are configured to PXE boot, the work is done. Make sure they are connected to each other through a network switch or a router, and you could technically let RIP.

Configuring DVD::RIP

You have to do some work on DVD::RIP to work. One is to ensure all the dependencies are satisfied, and the section part is to configure the different nodes.

Check the dependencies:

Open up DVD::RIP, it is normally under Applications->Sound & Video

In the DVD::RIP menu select Edit->Preferences

Under the Basic Settings tab, ensure your Default Data Base Directory and Default Directory for .rip Project Files are both set to the same as what you have specified previously, namely /home/yourusername/dvdrip-data. You may have to create the directory if it does not exist yet.

Select the Commands tab, and change xine to gxine for the STDIN player command

Once all the above changes have been made, click the CHECK ALL SETTINGS button and ensure there are no errors displayed. If no errors are displayed, click on OK to close the Preferences window.

Setting up the Cluster

The moment you all have been waiting for is here. In the DVD::RIP menu select Cluster->Cluster Control. You may sometimes get an error message saying “Can’t start local master daemon on port 28646. Execute the dvdrip-master program by hand to see why it doesn’t run.”

This is not a crisis, and all you have to do is to open a terminal window and type:

Code:

dvdrip-master 2

and you are good to go. A new window will open “dvd::rip Cluster Control”, which is the main cluster control screen you will use to configure, control and transcode projects with, using your cluster.

If your Cluster master daemon started automatically, you can skip the next step. For those who had to manually start the daemon, select Master->Connect Master Daemon here from the menu.

Setting nodes in DVD::RIP cluster

Once that is out of the way, time to add some nodes. Now assume we have two nodes, all ready PXE booted and somewhere on your ethernet. As we have previously set the prefix to our nodes as ClusterNode, and we set our IP address range for our nodes from 192.168.100.11 to 192.168.100.22 (12 clients, see above), your clusters will be named from ClusterNode111 to ClusterNode122. The first two nodes will be ClusterNode122 and ClusterNode121 respectively.

Select Node->Add Nodes from the top menu, and enter the following data:

Code:

  Name                                     = ClusterNode122
  Hostname                               = ClusterNode122
  Local Data Directory                = /home/yourusername/dvdrip-data
  Username to connect with SSH = yourusername

Click on the TEST SETTINGS button and a window should open asking you if you are sure about your marriage, ssh connections etc etc. In the entry box type “yes” and press Enter, and your node should be added at this stage.

If there is a problem ensure you have done the ssh settings as described above, and the data directory has been created. To ensure your node is active, open a terminal window and type:

Code:

ping 192.168.100.22

for ClusterNode122 etc. If you see a ping response, you know the node is alive and kicking.

Repeat this for all your other nodes

If everything works as expected, you should see all your nodes as idle, like you, waiting for something to happen.

From here on rip your movie as normal with DVD::RIP. Once they are ripped, and you are ready to transcode, select the ADD to CLUSTER button, switch to your Cluster Control screen, select the project and click on the START PROJECT, and your cluster should fly off and transcode in warp speed.

Performance

I have been able to rip and transcode a 2.5h movie to xvid in 55 minutes, and a 1.5h movie in 26 minutes, so you can see the benefits clearly. For an excellent guide to Linux DVD conversion see http://www.bunkus.org/dvdripping4lin…index.html#toc

Hope this helps all of you to use those spare machines lying around to convert a DVD quickly when you need to. I am able to run 10 nodes from my main Ubuntu box, without any impact, and as I do not use the main box as a node (which you can do just as you would for any other node) it does not impact on my normal use of that PC (except whilst ripping the DVD initially)

Good luck and I hope it works for you.

Last edited by thebigfatgeek; October 29th, 2008 at 01:28 PM..

Install ffmpeg for alembik transcoding server with 51.40.4 on ubuntu

mufumbo — Thu, 07 May 2009 12:31:07 +0000

I find the alembik transcode server a cool piece of software, but I found lots of caveats when started to use it.

The most difficult part is that alembik supports only two versions of libavcodec for ffmpeg, so you must compile it from the sources, since they are quite old.
That happens because ffmpeg changes the parameters from version to version and the alembik team had choosen both 51.40.4 and 51.48.0 to be supported. I think that in the future it will be possible to have an external “.properties” file, so I hope that it will not be necessary to compile ffmpeg by hand, every time.

I’ve used this references:
http://sourceforge.net/mailarchive/message.php?msg_name=48A176CC.8080300%40kimiasol.com (here i’ve found the rpm of a ffmpeg with libavcodec version 51.40.4 to download)
http://juliensimon.blogspot.com/2008/12/howto-compiling-ffmpeg-x264-mp3-xvid.html (this post teach how to install libamr from mediubuntu repository)

install both amr, you can install also from mediubuntu repo.

wget http://ftp.penguin.cz/pub/users/utx/amr/amrnb-6.1.0.4.tar.bz2
wget http://ftp.penguin.cz/pub/users/utx/amr/amrwb-7.0.0.1.tar.bz2
# apt-get install g++ nasm zlib1g-dev libx264-dev libfaad-dev faad liba52-0.7.4-dev libgsm1-dev libmp3lame-dev \
libtheora-dev libvorbis-dev libxvidcore4-dev libfaac-dev libimlib2-dev libfreetype6-dev

Use an aplication to convert the rpm to tar and decompress it (with alien or something else):

# wget http://dag.wieers.com/rpm/packages/ffmpeg/ffmpeg-0.4.9-0.9.20070530.rf.src.rpm

# ./configure --enable-libmp3lame --enable-libogg --enable-libvorbis --enable-libamr-nb --enable-libfaad --enable-libfaac \
--enable-libgsm --enable-xvid --enable-x264 --enable-liba52 --enable-liba52bin --enable-pp --enable-shared --enable-pthreads \
--enable-libtheora --enable-gpl --disable-strip --enable-libfaadbin --prefix=/usr/local --enable-swscaler --enable-libamr-wb

Install it as a package or with “make && make install” and settup alembik with “ffmpeg.version=51.40.4″

Getting video on the internet right

Remco — Sat, 02 May 2009 21:44:08 +0000

In the past two years that I’ve been working in transcoding at Joost, I have learnt a lot about what works, and what doesn’t, when it comes to putting video on the internet, and I have also seen that in many cases, the best thing to do is go back to the source. This is too much to fit into one guide, so today I will just introduce you to some of the characteristics of great looking video, and in future installments I will explain how to get individual things right using free (and possiby open source tools).

On quality

Most websites will compress video a lot, and encode video to a smaller frame size. Nevertheless it’s a bad idea to send low quality files to aggregrators; each encode will cause more quality loss, and if an encoder tries to encode a low-quality blocky video, the encoder will spend a lot of bits encoding the artifacts in the file you delivered instead of the actual desired picture. So just send the high quality, full size (be it 480p, 720p, or 1080p) and leave the dealing with scaling to the aggregators. They have their systems set up to do that anyhow.

File formats

I recommend two file formats, when preparing video for delivery. Either good old MPEG-2, the workhorse of the video industry, or mp4, with H.264 video and AAC audio. You almost cannot go wrong if you use the following bitrates for these formats:

MPEG-2, SD: 8 Mbps
H.264, 480p: 3 Mbps
H.264, 720p: 8 Mbps
H.264, 1080p: 20 Mbps

I must admit that I have relatively little knowledge of Windows Media, but while it has been common for streaming video for a while, it has lost ground to H.264 video these days, and it is not a native format for most video editors, so I can’t recommend it. Generally I recommend staying away from Quicktime if at all possible. Although modern Quicktime files generally will work just fine because it contains more or less the same things as a mp4 with H.264 and AAC, I have seen at least one occurence where there were cards (think a couple of slides) at the beginning of the video which were not part of the video track, causing ffmpeg based players and encoders (think Youtube and VLC) to not display things as expected.

Fill the frame

In online video you only have a limited amount of pixels, it’s a shame to waste about of a third of the pixels available to encode just black because your content is letterboxed, pillarboxed, or worst of all: windowboxed. Letterboxed content looks like this:

Letterboxed video, for in-depth explanation on letterboxing, see the wikipedia entry

Make sure there are no interlacing artifacts

One of the biggest killers for online video are interlacing artifacts. Interlacing artifacts are horizontal stripes, that can mostly be seen during fast horizontal movements, when the camera gets panned, or at very quick transitions. They are caused by the fact that back in the day televisions did not build up the picture progressively, but first rendered the odd lines, and then the even. Handbrake’s guide on interlacing is pretty good, and it is also a great tool to remove it.

Example of video with interlacing artifacts

In almost all cases I have seen of interlacing artifacts, these have been caused by issues in the video that has been uploaded. The two ways you can get interlacing artifacts in your video are:

Encoding a video that was interlaced to progressive without running a deinterlace filter.
Resizing a interlaced video without deinterlacing it first.

As far as I can tell there is no real decent way to remove interlacing artifacts once the damage is done, so if you ever have issues with this, generally the only way to go is to move back to the step when your video was still interlaced, and fix things at that stage.

Saas-Watch: 'HD Cloud' combines HD and clouds.

Rich — Thu, 16 Apr 2009 10:11:23 +0000

Cloud computing and Software-as-a-service needs more interesting use-cases! It seems that most examp

Transcoding is a bitch

erator — Mon, 02 Mar 2009 01:34:01 +0000

What is transcoding? Well, according to Wikipedia, “Transcoding is the direct digital-to-digital conversion of one encoding to another. This is usually done to incompatible or obsolete data in order to convert it into a more suitable format.”

In my case, I’m transcoding my family videos that were taken with our point-n-shoot camera. So here I am transcoding all the videos I took over the past year, in batches of files organized per month, then running my dandy little tool I wrote about yesterday that helps keep the Created and Modified dates intact.

But I wonder, there has to be a much better way to do this! So I dug around VLC media player, and noticed it had a “Save As/Convert” option in the file menu. I tried using it and converting a couple of files, but the non-helpful UI led me nowhere after I caused the program to crash a couple of times.

Anyone out there know how a better way? I’m running out of patience!

Transcoding for Security Video

Paul Galburt — Sat, 24 Jan 2009 14:15:32 +0000

Technology leaps in video security are often fueled by developments adopted from large dollar markets like entertainment. The use of transcoding and format down sampling in the DynaView SOC Video Management System are good examples of this approach. This article offers a good discussion of the power and value of these techniques.

The Evolution of Transcoding

For live video streams used in security and surveillance applications on demand transcoding and format conversion offer substantial reductions in wide area network utilization while requiring practical levels of server and router power.

Badaboom's CPU utilization issue explained

theovalich — Fri, 24 Oct 2008 12:00:42 +0000

After the first look here, I managed to again speak with Sam and Mike of Elemental fame, who got bac

Time for Episodic

js — Wed, 22 Oct 2008 20:00:44 +0000

Episodic

A quick shout out to those hard working fellers at Episodic! I am thoroughly impressed. I keep telling people here about the app — about the great interface and ease of use. Hard to believe the application is still in the embryonic stage.

I have a little experience in the digital video world from my years at Virage (consumed, digested, and shit out by Autonomy during the dot.com bust). Virage was focused on the business-end of video deployment, but there was one serious flaw with the business model: bandwidth. There wasn’t a whole lot of bandwidth to go around, and so sharing video or using video in your business was pretty much out of the question. Eight years later, of course, bandwidth is ubiquitous for industrialized nations. Codecs have caught up, too.

The difficulty in the online video market is that it’s flooded with companies trying to define it. From the Netflix and Blockbusters, the Hulas and Youtubes, and the Brightcove and Mavens (now Yahoo). Everyone wants a piece of the action. But few, I think, have got right. Episodic is one of them.

Easy to upload and transcode video
Easy to create episodes
East to insert interstitials
Easy to syndicate onto other networks (via TubeMogul)
Great metrics

That’s not to say that I don’t have my list of feature requests! And while I know I won’t get everything, I can take heart that the folks there care — a lot — about their customers.

Hey, if you guys are giving out stock options, mind tossing a few shares my way?

Make a Video Tutorial - Convert AVI to MP4 with MediaCoder

Mo Hax — Tue, 21 Oct 2008 18:37:50 +0000

Another segment in making a video tutorial for free. We use MediaCoder to make an MP4 from the AVI file captured with CamStudio and altered with VirtualDub. MP4 format is suitable for web site distribution, playing on iPod, and iTunes subscriptions.

Sobre a hospedagem de conteúdo de terceiros

mlonlinegeneration — Thu, 11 Sep 2008 13:04:25 +0000

Um juiz de um tribunal federal da Califórnia, Howard Lloyd, sentenciou a favor do Veoh, um site norte-americano de vídeos, em um caso em que este foi processado por suposta violação a direitos de propriedade da empresa de “entretenimento adulto” IO Group. O caso completo pode ser acessado aqui. Além do impacto óbvio para o Veoh, a decisão favorece também o YouTube, seu primo mais conhecido, e todos os sites que hospedam vídeos. De uma forma mais ampla, a decisão é importante para provedores e sites em geral que hospedam conteúdo de terceiros, tais como publicadores de blogs, fotologs, vídeos, perfis, comunidades e comércio eletrônico.

A decisão baseou-se no DMCA, Digital Millenium Copyright Act, legislação que protege quem hospeda conteúdo de terceiros, desde que estes sigam algumas regras, tais como informar seus usuários da proibição de inserir conteúdo protegido por direitos, atender rapidamente as solicitações de remoção de conteúdo indevido e banir os usuários infratores.

Para os sites que são plataformas de conteúdo para os usuários, ser responsabilizado por tudo que está em suas fronteiras é complicado e arbitrário. A decisão é mais enfática ainda ao dizer os sites não são obrigados a checar a priori todos os vídeos que hospedam.

A solução, neste casos, é dispor de mecanismos de denúncia para que de forma colaborativa (internautas e empresas), os conteúdos indevidos sejam notificados aos provedores e posteriormente, mediante denúncia e investigação, sejam removidos.

De novo, reforço, no MercadoLivre temos o Programa de Proteção à Propriedade Intelectual que funciona como um dispositivo para os detentores de propriedade intelectual nos alertarem caso encontrem alguma irregularidade ligada aos anúncios do site. Também temos um link para que qualquer internauta que identifique um conteúdo impróprio possa nos fazer uma denúncia. A partir daí, temos um processo de tomada de providências, culminando com a remoção de conteúdo indevido.

Veja mais na web
Transcoding Is Not A Crime, Says Court In Veoh Porn Case (Fonte TechCrunch)

OTOY's cinematic rendering on your browser

consultaglobal — Mon, 08 Sep 2008 03:10:04 +0000

“A mobile phone may have to rescale and halftone the image to display it properly on a

Badaboom - GPU Accelerated Video Transcoding

only99cents — Sat, 06 Sep 2008 12:02:50 +0000

So earlier yesterday I checked out a new program called Badaboom. Right now it’s in beta 2, it’s a video encoder/transcoder that uses Nvidia GPU’s with CUBA enabled cores to encode video files and DVD’s. I couldn’t find a good DVD worth testing it out on, and I tried it with American History X, but it was rejected do to copywrite protection, ah the glory in that.

Anyway, the program is something to check out, in the future. Right now it may work fine for you, but what I need it for, it just won’t do. It has a large array of video templates including, Xbox 360, Playstation 3, iPod’s, HD, PSP, and a few others I can’t remember off the top of my head.

So head over to Nvidia’s forcewithin here, and download Beta 2, it’s a evaluation copy and ends september 30th, but im sure theres a work around for that.

Scalable Video Coding (SVC) smoothes MPEG4 h.264

Dave — Sat, 06 Sep 2008 06:29:21 +0000

YouTube, YouTube, YouTube. They’re always making announcements, some guff, some not. But their latest annoucement – that they’ll be re-encoding their videos using an enhancement to the MPEG4 h.264 codec, called scalable video coding (SVC) – is seriously big news.

In simple terms, the idea behind encoding videos is you divide up the picture into blocks, and estimate the motion between the blocks. This allows you to store the difference between frames, and not the whole frame, which reduces the file size. MPEG4 h.264 is particularly wonderful at doing this, but when packets of information get delayed or lost while traveling through the Internet, you get small glitches in the picture. SVC to the rescue! Those clever boffins at Fraunhofer have devised a way to enhance the existing codec to accommodate these packet drops. And the best news is that it seems we won’t need to change the players to make it work.

Hopefully it won’t be too long before we implement SVC in the Kensei encoder…