Linux: How to split large files into several parts

HJSplit

HJSplit for Linux is a freeware file splitter for Linux with full graphical user-interface.
Download ‘HJSplit’ from Freebytesoftware

Unix Split command

$ split -l 1000 my_file.txt     # Split the file named my_file.txt into several files with 1000 lines 
$ split -b n my_file.txt	# Split a file into pieces n bytes in size. 

Create, write and read a file and also print the first line of a file in ruby

Create a file: Open a file in write mode

ruby-1.9.2-p290 :058 > f = File.open(“my_file.txt”, “w”)
=>

Write something to that file

ruby-1.9.2-p290 :059 > f.write(“This is the first line of my file\n”)
=> 34

Close the file

ruby-1.9.2-p290 :060 > f.close
=> nil

Open the file for reading

ruby-1.9.2-p290 :061 > f = File.open(“my_file.txt”, “r”)
ruby-1.9.2-p290 :062 > f.read
=> “This is the first line of my file.\n”

Print the first line of a file

ruby-1.9.2-p290 :063 > lines = IO.readlines(“my_file.txt”)
ruby-1.9.2-p290 :063 > puts lines.first
This is the first line of my file.

A useful link about ruby File manupulation is given below

Open ods, xls, xlsx files in ruby by using ‘roo’

Install ‘roo’ gem
$ sudo gem install ‘roo’
take the ruby console
$ irb
for example take an xlxs file and try to open

ruby-1.9.2-p290 :003 > require ‘roo’
if you get an error like
/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:6: warning: already initialized constant VERSION
/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:7: warning: already initialized constant LIBPATH
/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:8: warning: already initialized constant PATH
LoadError: no such file to load — zip/zipfilesystem
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo/openoffice.rb:3:in `’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:71:in `’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from (irb):3
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `’
ruby-1.9.2-p290 :004 >

Then install the ‘rubyzip2’ gem

$ gem install rubyzip2

and open the file
ruby-1.9.2-p290 :003 > file = Excelx.new(“/home/prescience-toshiba/Documents/abhilash.raw.xlsx”)

output:
=> {[1, 1]=>”Name”, [1, 2]=>”Email Id”, [1, 3]=>”Alternate Number”, [1, 4]=>”Date of Birth”, [1, 5]=>”Mobile No.”, [1, 6]=>”Functional Area”, [1, 7]=>”Area of Specialization”, [1, 8]=>”Industry”, [1, 9]=>”Resume Title”, [1, 10]=>”Key Skills”, [1, 11]=>”Work Experience”, [1, 12]=>”Current Employer”, [1, 13]=>”Previous Employer”, [1, 14]=>”Current Salary”, [1, 15]=>”Level”, [1, 16]=>”Current Location”, [1, 17]=>”Preferred Location”, [1, 18]=>”Course(Highest Education)”, [1, 19]=>”Name”, [1, 20]=>”Email Id”, [1, 21]=>”Alternate Number”, [1, 22]=>”Date of Birth”, [1, 23]=>”Mobile No.”, [1, 24]=>”Functional Area”, [1, 25]=>”Area of Specialization”, [1, 26]=>”Industry”, [1, 27]=>”Resume Title”, [1, 28]=>”Key Skills”, [1, 29]=>”Work Experience”, [1, 30]=>”Current Employer”, [2, 1]=>”Rahel Robert”, [2, 2]=>”rtrobert@yahoo.com”, [……………………..

Ruby gem roo error: LoadError: no such file to load — zip/zipfilesystem

ruby-1.9.2-p290 :003 > require ‘roo’
/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:6: warning: already initialized constant VERSION
/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:7: warning: already initialized constant LIBPATH
/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:8: warning: already initialized constant PATH
LoadError: no such file to load — zip/zipfilesystem
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo/openoffice.rb:3:in `’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/roo-1.10.0/lib/roo.rb:71:in `’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require’
from (irb):3
from /home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `’
ruby-1.9.2-p290 :004 >

Instead of roo do

ruby-1.9.2-p290 :004 > require ‘roo/excel’

This works fine. If you want zip gem then install it.

$ gem install rubyzip2

Then do

ruby-1.9.2-p290 :001 > require ‘roo’

=> true

Useful commands and Usually using software’s installations in Fedora 15

Extract rpm files in Fedora


You can easily install rpm file with rpm -ivh

$ rpm -ivh file-package.name.rpm

Install Google chrome browser


$ sudo yum localinstall https://dl-ssl.google.com/linux/direct/google-chrome-stable_current_i386.rpm

Command for creating a ssh key


$ ssh-keygen -t rsa -C “your_email@youremail.com”

Install Adobe flash player


$ su –
## Adobe Repository 32-bit x86 ##
$ rpm -ivh http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm
$ rpm –import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

## Adobe Repository 64-bit x86_64 ##
$ rpm -ivh http://linuxdownload.adobe.com/adobe-release/adobe-release-x86_64-1.0-1.noarch.rpm
$ rpm –import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

Update repositories

$ yum check-update

Install Needed Packages and Adobe Flash Player 11

$ yum install flash-plugin nspluginwrapper alsa-plugins-pulseaudio libcurl

Install Chromium browser


$ su –
## Change directory to /etc/yum.repos.d/ ##
$ cd /etc/yum.repos.d/

## Get Chromium repo file ##
$ wget http://repos.fedorapeople.org/repos/spot/chromium/fedora-chromium-stable.repo
$ yum install chromium

Install VLC media player


$ su –
$ rpm -ivh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm
$ yum install vlc
$ yum install mozilla-vlc (optional)

Install Gimp


$ sudo yum install gimp

Install Office packages


$ sudo yum groupinstall “Office/Productivity”

Install Rails 3.1 ,Ruby, Mysql, Thinking Sphinx using RVM in Fedora 15

Download the fedora live CD from fedora website


Download Fedora live CD from Fedora Project Website. Install it in your machine.
After the installation goto Applications and search for ‘install to hard disk’ option. Install by chosing the custom option.
* When you try to install using yum and got an error like ‘your username is not in the sudoers file. This incident will be reported’ Then

Edit the sudoers file
To edit it, use the command

gedit /etc/sudoers then if it again shows ‘can’t open the file. You do not have the permissions necessary to open the file’ then

Log in as root (terminal) and type the following:

To get into root type $ su –
password:

then add your username to sudoers file:

$ echo ‘prescience-toshiba ALL=(ALL) ALL’ >> /etc/sudoers

Install RVM and Rails

First Install curl


$ sudo yum install curl

Install git


$ sudo yum install git-core

Install gcc


$ sudo yum install gcc

Install the following dependancies


$ sudo yum install readline-devel.i686
$ sudo yum install zlib
$ sudo yum install sqlite-devel
$ sudo yum install ruby-devel

Then Install RVM


$ bash < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer )

Add the following line to your .bash_profile file

[[ -s "/home/prescience-toshiba/.rvm/scripts/rvm" ]] && source "/home/prescience-toshiba/.rvm/scripts/rvm" # This loads RVM into a shell session

replace 'prescience-toshiba' with your username

For ruby 1.9.2 we want the packages it lists under the MRI & ree section. Let’s install those now.

$ sudo yum install bison openssl curl git-core libssl-dev vim libxml2-dev git-core subversion autoconf

Then type
$ rvm list known

Then install ruby 1.9.2


$ rvm install 1.9.2

Type
$ rvm list rubies

You can see ruby versions.

To list default ruby

$ rvm list default

You can change it by
$ rvm –default ruby-1.9.2-p290

Create Gemsets


$ rvm gemset create rails31
$ rvm gemset create rails31rc

$ rvm list gemsets

As a best practice, remember to always use one gemset per project*.
Installing Rails

Select the gemset for Rails 3.1
$ rvm use 1.9.2-p290@rails31rc

for default $ rvm use ruby-1.9.2-p290@rails31rc –default

Installing Rails 3.1


$ gem install rails -v “>=3.1.0.rc4”

If you get an error like

ERROR: Loading command: install (LoadError)
no such file to load — zlib
ERROR: While executing gem … (NameError)
uninitialized constant Gem::Commands::InstallCommand

then install zlib

$ rvm pkg install zlib

then

$ rvm remove 1.9.2
$ rvm install 1.9.2

If the error shows again do

$ sudo yum install zlib zlib-devel

then again do the following

$ rvm remove 1.9.2
$ rvm install 1.9.2

Install Mysql


$ sudo yum -y install mysql-server

See Mysql is running or not
$ sudo netstat -tap | grep mysql
If Yes then you can see like

tcp        0      0 localhost:mysql         *:*                     LISTEN      2556/mysqld

Also install mysql-devel
$ sudo yum install mysql-devel

Start Mysql

$ sudo /etc/rc.d/init.d/mysqld start
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following command:

$ sudo /usr/bin/mysqladmin -u root password ‘new-password’

$ mysql -u root -p
password:

mysql>

Possible errors while doing bundle install and its solutions


1)
checking for rb_thread_blocking_region()… yes
checking for mysql.h… no
checking for mysql/mysql.h… no

—–
mysql.h is missing. please check your installation of mysql and try again.

Install mysql-dev
$ sudo yum install mysql-devel

2)
Installing nokogiri (1.5.0) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/bin/ruby extconf.rb
checking for libxml/parser.h… no
—–
libxml2 is missing. please visit http://nokogiri.org/tutorials/installing_nokogiri.html for help with installing dependencies.
—–
*** extconf.rb failed ***
Install the following

$ sudo yum install libxml2-devel libxslt-devel

UPDATE: In Rails 4.2.5 (Ubuntu 14.04), I got the following similar error

Fetching: nokogiri-1.6.7.2.gem (100%)                                                           
Building native extensions.  This could take a while...                                         
ERROR:  Error installing rails:                                                                 
        ERROR: Failed to build gem native extension.                                            
                                                                                                
    current directory: /home/vagrant/.rvm/gems/ruby-2.3.0/gems/nokogiri-1.6.7.2/ext/nokogiri    
/home/vagrant/.rvm/rubies/ruby-2.3.0/bin/ruby -r ./siteconf20160311-22083-xt47t1.rb extconf.rb  
checking if the C compiler accepts ... *** extconf.rb failed ***                                
Could not create Makefile due to some reason, probably lack of necessary                        

Solving this install the following package
$ sudo apt-get install libgmp-dev
3)
Installing therubyracer (0.9.8) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/bin/ruby extconf.rb
creating Makefile

make
g++ -I. -I/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/libv8-3.3.10.2-x86-linux/lib/libv8/v8/include -I/home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/include/ruby-1.9.1/i686-linux -I/home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/include/ruby-1.9.1/ruby/backward -I/home/prescience-toshiba/.rvm/rubies/ruby-1.9.2-p290/include/ruby-1.9.1 -I. -D_FILE_OFFSET_BITS=64 -Wall -g -rdynamic -fPIC -O3 -ggdb -Wextra -Wno-unused-parameter -Wno-parentheses -Wpointer-arith -Wwrite-strings -Wno-missing-field-initializers -Wno-long-long -fPIC -o v8_date.o -c v8_date.cpp
make: g++: Command not found
make: *** [v8_date.o] Error 127

Install package ‘gcc-c++’ to provide command ‘c++’? [N/y]

$ sudo yum install gcc-c++

Possible errors while using rake command and its solutions


1)
/home/prescience-toshiba/.rvm/gems/ruby-1.9.2-p290@rails31rc/gems/rack-1.3.4/lib/rack/backports/uri/common_192.rb:53: warning: already initialized constant WFKV_
rake aborted!
no such file to load — openssl

2) If you get sprockets error like undefined method static_root
then upgrde your rails to 3.1.0.rc08

3) Started GET “/” for 127.0.0.1 at 2011-10-18 17:47:18 +0530 Mysql2::Error (Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)):

Then make sure that your system have /var/run/mysqld/mysqld.sock file exists. do $ ll /var/run/mysqld/mysqld.sock This will show the details. In my system the mysql.sock is in ‘/var/lib/mysql/mysql.sock’ So create a link to /var/run/mysqld/mysqld.sock from /var/lib/mysql/mysql.sock $ sudo ln -s /var/lib/mysql/mysql.sock /var/run/mysqld/mysqld.sock

$ sudo yum groupinstall development-tools development-libs
$ rvm remove 1.9.2
$ rvm cleanup all
$ rvm install 1.9.2
$ rvm use 1.9.2 –default

Install sphinx


let’s first search for the package

$ yum search sphinx

$ wget http://sphinxsearch.com/files/sphinx-0.9.9.tar.gz

$ ./configure
$ make
$ sudo make install

If shows Error like:

ERROR: cannot find MySQL include files.

Check that you do have MySQL include files installed.
The package name is typically 'mysql-devel'.

If include files are installed on your system, but you are still getting
this message, you should do one of the following:

Then You need to install mysql-devel in fedora
$ sudo apt-get install mysql-devel
OR in Ubuntu
$ $ sudo aptitude install libmysqlclient15-dev

Type
$ search
to see sphinx is installed

how to start mysql at boot time


$ cd /etc/rc.d/rc3.d
$ ln -s /etc/init.d/mysql S98mysql

How does it work? /etc/rc.d/init.d contains scripts to start and stop services on your computer. By creating a symlink to mysql startup script in /etc/rc.d/rc3.d you’re telling your system to run it during runlevel 3 startup, which is your regular login. You can also start and stop your services manually using files in /etc/init.d directly, for example:

$ /etc/init.d/mysql start
$ /etc/init.d/mysql stop

Installing rails 3.1 and ruby 1.9.2-p180 using rvm and unicorn setup

Install rvm

Open your .bashrc file.
And replace the following line
[ -z “$PS1” ] && return
with
if [[ -n “$PS1” ]]; then
Now add this to the last line of the file
if [[ -s $HOME/.rvm/scripts/rvm ]] ; then source $HOME/.rvm/scripts/rvm ; fi

fi
This loads RVM into a shell session.

If already have rvm then
Update rvm

$ rvm get latest

if this shows error like ‘downloaded does not match it’s md5 checksum’

then run

$ rvm get head
$ rvm reload
$ rvm get latest

For ruby 1.9.2 we want the packages it lists under the MRI & ree section. Let’s install those now.

$ sudo aptitude install build-essential bison openssl libreadline5 libreadline-dev curl git-core zlib1g zlib1g-dev libssl-dev vim libsqlite3-0 libsqlite3-dev sqlite3 libreadline-dev libxml2-dev git-core subversion autoconf

Setting up Ruby

Do rvm list known to know all ‘Ruby implementation made available through RVM’

$ rvm list known

Then install latest stable release of ruby

$ rvm install 1.9.2

You can see the ruby versions installed and currently using ruby version by

$ rvm list rubies

To list default ruby

$ rvm list default

You can change it by

$ rvm use ruby-1.9.2-p180

for default : rvm –default 1.9.2-p180

Using Gemsets

Install rails stable release and then rails 3.1
Start by creating our gemsets

$ rvm gemset create rails307 rails31

See available gem sets by

$ rvm gemset list

If any confusion with rails31 gemset delete it

$ rvm gemset delete rails31

And create rails31rc

$ rvm gemset create rails31rc

As a best practice, remember to always use one gemset per project*.

Installing Rails

Now we have multiple gem set installed. Now set the different gem set for ruby versions.

$ rvm use 1.9.2-p180@rails307

Install Rails 3.0.7

$ gem install rails [-v 3.0.7] [–no-rdoc]

Select the gemset for Rails 3.1

$ rvm use 1.9.2-p180@rails31rc

for default $ rvm use 1.9.2-p180@rails31rc –default

Install Rails 3.1

$ gem install rails -v “>=3.1.0rc”

if not found

$ gem install rails -v “>=3.1.0.rc4”

rvmrc file

Switching from one project to another, from a client to a personal project, from testing the release candidate to developing using the latest stable version, always having to manually switch from using a gemset to another can impact productivity. The project .rvmrc files can increase the development speed by setting up our project’s ruby environment when we switch to the project root directory.

The rule of thumb here is to use a .rvmrc file for each project, for both development and deployment.*

.rvmrc file :

rvm –rvmrc use 1.9.2-p180@rails31rc

if you go to your app folder ‘$ cd myapp’ then you can see like





There are three types of .rvmrc files
1) system : on /etc/rvmrc
2) user : $HOME/.rvmrc
3) Project : .rvmrc

Create a new rails App

$ rails new myapp

with mysql as database

$ rails new myapp -d mysql
$ rails server -p4000

If u get error like

“”” home/user/.rvm/gems/ruby-1.9.2-p180@rails31rc4/gems/execjs-1.1.2/lib/
execjs/runtimes.rb:43:in `autodetect’: Could not find a JavaScript
runtime. See https://github.com/sstephenson/execjs for a list of
available runtimes. (ExecJS::RuntimeUnavailable) “””

Then you need a javascript engine to run your code…

try installing rubyracer (Embeds the V8 Javascript interpreter into Ruby.)

in your Gemfile add

gem ‘therubyracer’
and do bundle install

Unicorn Installation

$ rvm use default # Just to make sure
$ gem install unicorn

Start unicorn server

$ unicorn_rails