|
|
|
|
Thread title: which language do i need to use for processing 500gb of data? |
|
|
|
|
|
Thread tools
Search this thread
Display Modes
|
|
06-03-2012, 08:58 AM
|
#1
|
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
Posts: 41
|
which language do i need to use for processing 500gb of data?
i have a server filled with 500GB of docx, pptx, html files, pdfs etc. I wanted to create a search engine where all these files will b crawled nd indexed! I wud like to know which programming language should i rely on? Am an expert php developer! I found php works good, but it takes alot of time to crawl nd process each file! Any solution or suggestions? Do i need to code in python, perl, c, c++, java or in ruby on rails?
|
|
06-03-2012, 01:37 PM
|
#2
|
Status: Geek
Join date: Apr 2006
Location: Denver, CO
Expertise: Software
Software: Chrome, Notepad++
Posts: 6,894
|
Nothing is going to search half a terabyte quickly. Your algorithms are going to matter more than the language you do it in. C will be the fastest but if you just iterate though a directory it's all going to be slow. You're going to need to index these files and search through that.
|
|
06-03-2012, 02:57 PM
|
#3
|
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
Posts: 41
|
i will be creating a crawler to index these files, but i need a programming language that crawls and indexes these files realy fast. The searchng is being done using php which i dont mind changing if there is any other language whch can do better than php! Btw if i were to use C language, how would i execute the program on a server and how wil i be able to index these files on a MySQL database?
|
|
06-03-2012, 05:45 PM
|
#4
|
Status: Geek
Join date: Apr 2006
Location: Denver, CO
Expertise: Software
Software: Chrome, Notepad++
Posts: 6,894
|
PHP has command line access (shell_exec()) so you can run the programs you make, there are also MySQL libraries for C. If you aren't familiar with Linux programming doing it C will be no trivial task; everything is hard in C..
What you need to do is see which parts of your program are taking the longest and optimize them.
|
|
06-04-2012, 05:34 AM
|
#5
|
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
Posts: 41
|
i just found that c language is pretty fast and it would be great if u could show me an example of executing a c program using php! I tried googling it, but nothng seemed to work.
|
|
06-04-2012, 06:54 AM
|
#6
|
Status: Request a custom title
Join date: Feb 2005
Location:
Expertise:
Software:
Posts: 1,159
|
|
|
06-04-2012, 08:01 AM
|
#7
|
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
Posts: 41
|
i dont see a C program inside it! If am nt wrong, that has got to be linux coding, ryt?
|
|
06-04-2012, 02:32 PM
|
#8
|
Status: Request a custom title
Join date: Feb 2005
Location:
Expertise:
Software:
Posts: 1,159
|
Originally Posted by unikorndesigns
i dont see a C program inside it! If am nt wrong, that has got to be linux coding, ryt?
|
C programs are compiled into executables.
|
|
06-04-2012, 03:00 PM
|
#9
|
Status: Geek
Join date: Apr 2006
Location: Denver, CO
Expertise: Software
Software: Chrome, Notepad++
Posts: 6,894
|
Originally Posted by unikorndesigns
i just found that c language is pretty fast and it would be great if u could show me an example of executing a c program using php! I tried googling it, but nothng seemed to work.
|
C needs to be compiled into a program. If you're in a linux setup you're going to need to learn how to use the GCC compiler. There's tons of resources on google. There's also IDEs out there but I usually just use Vim and GCC on the command line.
|
|
06-04-2012, 04:01 PM
|
#10
|
Status: Junior Member
Join date: May 2011
Location:
Expertise: Web Designing and Development
Software: Photoshop, Dreamweaver and etc
Posts: 41
|
oh ok! I am using Wampserver! Is there a way to do for windows setup! I had tried google but the results did not seem to work, i suppose its because of da windows setup! Any soln that wud work with windows?
|
|
|
|
|
|
|
|
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
|
|
|
|