Any SQL wizard can help? Reformat the input file and transfer into SQL server

I am trying to transfer 200 txt files into SQL server by using query analyzer.
The command is 'Bulk insert [tableName] from 'path\filename.txt'
However, I need to read and modifiy the txt file.
I am new to SQL server but I believe there must be some one who is a wizard can do what I want easily.

Thank you for the help in advance!

Here is the raw data layout, which is comma delimited.
BDate 1/1/1990 BDate 1/1/1990 BDate 1/1/1990 BDate 1/1/1990
Edate 1/1/2005 Edate 1/1/2005 Edate 1/1/2005 Edate 1/1/2005
Fq D Fq D Fq D Fq D
Date R P M E D Date R P M E D Date R P M E D Date R P M E D
1/1/90 1 2 3 4 5 1/1/90 2 3 4 5 6 1/1/90 3 4 5 6 7 1/1/90 4 5 6 7 8
2 3 4 5 6 1 2 3 4 5 3 4 5 6 7 6 7 8 9 1
1/1/05 ..... 1/1/05 ... 1/1/05 .... 1/1/05 ....

This is the desired output after load into the table, which is tacking each repeating block on top of each other.
Date R P M E D
1/1/90 1 2 3 4 5
2 3 4 5 6
1/1/05 .....
1/1/90 2 3 4 5 6
2 3 4 5 6
1/1/05 .....
1/1/90 3 4 5 6 7
3 4 5 6 7
1/1/05 .....
1/1/90 4 5 6 7 8
6 7 8 9 1
1/1/05 ....."I am trying to transfer 200 txt files into SQL server by using query analyzer."
--DTS might be more appropriate.

"I am new to SQL server but I believe there must be some one who is a wizard can do what I want easily."
--Faith is a powerful thing.

"Here is the raw data layout, which is comma delimited."
--What you posted is not comma delimited.

"This is the desired output after load into the table, which is tacking each repeating block on top of each other."
--You are going to need to load this data into a staging table and normalize it before loading into your production tables. The process will be complex, involving several passes through the data.

If at all possible, try to get your source data in a better format. Practically any other format would be preferable to what you posted.|||Blindman,
Thank you for your reply.
You are right... I forgot to put "," in my sample file layout.
I am using another source provider to request time series in excel. This is the most efficient way I can utilize excel ability (256 columns and over 65,000 rows). That's why the raw data layout looks wired. However, I have to stick to it.

I was thinking to load these files into a table to normalize but I am not sure if I know SQL well enough to say this is the best solution. I think I got the answer from you.

What is staging db. I assume it is one of defualt DB in in enterprise manager, however, I did not see it. Or this is the name you gave?

Thank you again for the help.
Shiparsons|||Not "Staging DB". "Staging TABLE."

A staging table is basically an table that has the same structure as your input data, with additional columns added as needed to keep track of records as they are being processed. I always add an "Imported" column that defaults to getdate(), and an ImportErrors column that I populate as necessary during processing.

Your staging table should have no Primary Keys or constraints (unless you add a surrogate PKey for processing...), so that your import process never fails because the data does not match what is expected.

Once the data is in the staging table you cleans it and make sure it satisfies all the business rules required by your production tables. Then you make as many passes through the staging table as necessary to update the various production tables it feeds, starting with the top-level tables.|||Thank you for the explanation.
What datatype I should use when I create my staging table? I assume this is nonconstraints type since my raw data contains text, datetime, and float.

Thank you,
Qing|||You should try to match the datatype to the type of the data being entered, though some people just make all staging table columns varchar by default. I don't do this, as a rule, but you may have no other choice since your import file is actually a mix of different layouts. String fields are the only column types that will accept any input type.|||Blindman,
Thank you for the help.

I will try.

shiparsons

any order

Hi,
Is there any sort order if we don't use order by? for example,
select * from tablename
what is sort order? each time, I get the same records with the same order,
any mystery?As far as the I know, the order was the same as you add the data.
You can change the order bu add "order by" statement , I think you can get
whole information from BOL|||Wei
I don't think that order depends on insertion. (unless you have clustered
index on this column)
"Wei Ci Zhou" <weicizhou@.hotmail.com.discuss> wrote in message
news:O9uMAcW6DHA.2380@.TK2MSFTNGP10.phx.gbl...
> As far as the I know, the order was the same as you add the data.
> You can change the order bu add "order by" statement , I think you can get
> whole information from BOL
>|||Hi ,
If you have a clusted index in that table database will be ordered based on
the Index key. So incase if you have a clustered index the
"select * from tablename" will return the result set based on the clustered
index key order.
If you donot have a clusted index the result set will be reurned based on
the way you inserted.
Thanks
Hari
MCDBA
"John" <spam@.spam.com> wrote in message
news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> Hi,
> Is there any sort order if we don't use order by? for example,
> select * from tablename
> what is sort order? each time, I get the same records with the same order,
> any mystery?
>|||records are sorted in ascending order by default.
"John" <spam@.spam.com> wrote in message
news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> Hi,
> Is there any sort order if we don't use order by? for example,
> select * from tablename
> what is sort order? each time, I get the same records with the same order,
> any mystery?
>|||Sorry
> I don't think that order depends on insertion. (unless you have clustered
> index on this column)
I meant if you don't have a clustered index on the column it depends on
insertion.
"Uri Dimant" <urid@.iscar.co.il> wrote in message
news:#JxgijW6DHA.3548@.TK2MSFTNGP11.phx.gbl...
> Wei
> I don't think that order depends on insertion. (unless you have clustered
> index on this column)
>
> "Wei Ci Zhou" <weicizhou@.hotmail.com.discuss> wrote in message
> news:O9uMAcW6DHA.2380@.TK2MSFTNGP10.phx.gbl...
> > As far as the I know, the order was the same as you add the data.
> > You can change the order bu add "order by" statement , I think you can
get
> > whole information from BOL
> >
> >
>|||The first reply you made me shocked :)|||Incorrect, I'm afraid. See my other post.
--
Tibor Karaszi, SQL Server MVP
Archive at: http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
"harsh" <harshalmistry@.hotmail.com> wrote in message
news:%23RaX%23pW6DHA.2568@.TK2MSFTNGP10.phx.gbl...
> records are sorted in ascending order by default.
>
> "John" <spam@.spam.com> wrote in message
> news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> > Hi,
> >
> > Is there any sort order if we don't use order by? for example,
> >
> > select * from tablename
> >
> > what is sort order? each time, I get the same records with the same order,
> > any mystery?
> >
> >
>|||> If you have a clusted index in that table database will be ordered based on
> the Index key.
Not necessarily. See my other post.
--
Tibor Karaszi, SQL Server MVP
Archive at: http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
"Hari Prasad" <hari_prasad_k@.hotmail.com> wrote in message
news:uPL7gkW6DHA.2628@.TK2MSFTNGP10.phx.gbl...
> Hi ,
> If you have a clusted index in that table database will be ordered based on
> the Index key. So incase if you have a clustered index the
> "select * from tablename" will return the result set based on the clustered
> index key order.
> If you donot have a clusted index the result set will be reurned based on
> the way you inserted.
>
> Thanks
> Hari
> MCDBA
> "John" <spam@.spam.com> wrote in message
> news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> > Hi,
> >
> > Is there any sort order if we don't use order by? for example,
> >
> > select * from tablename
> >
> > what is sort order? each time, I get the same records with the same order,
> > any mystery?
> >
> >
>|||> I meant if you don't have a clustered index on the column it depends on
> insertion.
Not correct. I'm afraid. See my other post.
--
Tibor Karaszi, SQL Server MVP
Archive at: http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
"Uri Dimant" <urid@.iscar.co.il> wrote in message news:%23qCR2tW6DHA.1936@.TK2MSFTNGP12.phx.gbl...
> Sorry
> > I don't think that order depends on insertion. (unless you have clustered
> > index on this column)
> I meant if you don't have a clustered index on the column it depends on
> insertion.
>
> "Uri Dimant" <urid@.iscar.co.il> wrote in message
> news:#JxgijW6DHA.3548@.TK2MSFTNGP11.phx.gbl...
> > Wei
> > I don't think that order depends on insertion. (unless you have clustered
> > index on this column)
> >
> >
> >
> > "Wei Ci Zhou" <weicizhou@.hotmail.com.discuss> wrote in message
> > news:O9uMAcW6DHA.2380@.TK2MSFTNGP10.phx.gbl...
> > > As far as the I know, the order was the same as you add the data.
> > > You can change the order bu add "order by" statement , I think you can
> get
> > > whole information from BOL
> > >
> > >
> >
> >
>|||A table, is in the relation model, an unordered set of rows. Imagine your rows being notes on
paper-slips and you throw them into a bag. The back is shaken from time to time. Now. try to pull
the paper-slips out of this bag "in order". The term "order" doesn't make sense in the relational
model.
To be more specific, if you don't have order by, the optimizer is free to return the rows in any
order. It will pick the order it finds most cost-effective.
It doesn't have to be in the same sequence as the rows are inserted (read about IAM page etc in the
physical database chapter in Books Online). That is regardless whether you have a clustered index or
not.
It doesn't have to be in the same sequence as the clustered index. The index can be heavy
fragmented, so the optimizer can decide to instead of jumping back and forth on the disk, it will
scan the file in physical order, according to the IAM page.
--
Tibor Karaszi, SQL Server MVP
Archive at: http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
"John" <spam@.spam.com> wrote in message news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> Hi,
> Is there any sort order if we don't use order by? for example,
> select * from tablename
> what is sort order? each time, I get the same records with the same order,
> any mystery?
>|||Tibor
I have just finished to read the article, you are absolutely right. Sorry
for giniven incorrect information.
"Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote in
message news:OS1xRJX6DHA.2712@.tk2msftngp13.phx.gbl...
> A table, is in the relation model, an unordered set of rows. Imagine your
rows being notes on
> paper-slips and you throw them into a bag. The back is shaken from time to
time. Now. try to pull
> the paper-slips out of this bag "in order". The term "order" doesn't make
sense in the relational
> model.
> To be more specific, if you don't have order by, the optimizer is free to
return the rows in any
> order. It will pick the order it finds most cost-effective.
> It doesn't have to be in the same sequence as the rows are inserted (read
about IAM page etc in the
> physical database chapter in Books Online). That is regardless whether you
have a clustered index or
> not.
> It doesn't have to be in the same sequence as the clustered index. The
index can be heavy
> fragmented, so the optimizer can decide to instead of jumping back and
forth on the disk, it will
> scan the file in physical order, according to the IAM page.
> --
> Tibor Karaszi, SQL Server MVP
> Archive at:
http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
>
> "John" <spam@.spam.com> wrote in message
news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> > Hi,
> >
> > Is there any sort order if we don't use order by? for example,
> >
> > select * from tablename
> >
> > what is sort order? each time, I get the same records with the same
order,
> > any mystery?
> >
> >
>|||Tibor.
Greate Poster, let me learn many thing.
I do not think you are right after you poster, but after read the
information on BOL. Yes you are right.|||>The term "order" doesn't make sense in the relational
> model.
I understand that the order and PHYSICAL placement of the data is not a
concern in the relational model but surely the RM addresses returning
ordered data? Isn't this called sort sets in the RM?
--
Ray Higdon MCSE, MCDBA, CCNA
--
"Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote in
message news:OS1xRJX6DHA.2712@.tk2msftngp13.phx.gbl...
> A table, is in the relation model, an unordered set of rows. Imagine your
rows being notes on
> paper-slips and you throw them into a bag. The back is shaken from time to
time. Now. try to pull
> the paper-slips out of this bag "in order". The term "order" doesn't make
sense in the relational
> model.
> To be more specific, if you don't have order by, the optimizer is free to
return the rows in any
> order. It will pick the order it finds most cost-effective.
> It doesn't have to be in the same sequence as the rows are inserted (read
about IAM page etc in the
> physical database chapter in Books Online). That is regardless whether you
have a clustered index or
> not.
> It doesn't have to be in the same sequence as the clustered index. The
index can be heavy
> fragmented, so the optimizer can decide to instead of jumping back and
forth on the disk, it will
> scan the file in physical order, according to the IAM page.
> --
> Tibor Karaszi, SQL Server MVP
> Archive at:
http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
>
> "John" <spam@.spam.com> wrote in message
news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> > Hi,
> >
> > Is there any sort order if we don't use order by? for example,
> >
> > select * from tablename
> >
> > what is sort order? each time, I get the same records with the same
order,
> > any mystery?
> >
> >
>|||As far as I know, RM does not have this concept, as this would mean that RM defines other concepts
than relations. I might be wrong, of course and I welcome (as always :-) ) pointers etc...
--
Tibor Karaszi, SQL Server MVP
Archive at: http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
"Ray Higdon" <sqlhigdon@.nospam.yahoo.com> wrote in message
news:eNG0m9X6DHA.2380@.TK2MSFTNGP10.phx.gbl...
> >The term "order" doesn't make sense in the relational
> > model.
> I understand that the order and PHYSICAL placement of the data is not a
> concern in the relational model but surely the RM addresses returning
> ordered data? Isn't this called sort sets in the RM?
> --
> Ray Higdon MCSE, MCDBA, CCNA
> --
> "Tibor Karaszi" <tibor_please.no.email_karaszi@.hotmail.nomail.com> wrote in
> message news:OS1xRJX6DHA.2712@.tk2msftngp13.phx.gbl...
> > A table, is in the relation model, an unordered set of rows. Imagine your
> rows being notes on
> > paper-slips and you throw them into a bag. The back is shaken from time to
> time. Now. try to pull
> > the paper-slips out of this bag "in order". The term "order" doesn't make
> sense in the relational
> > model.
> >
> > To be more specific, if you don't have order by, the optimizer is free to
> return the rows in any
> > order. It will pick the order it finds most cost-effective.
> >
> > It doesn't have to be in the same sequence as the rows are inserted (read
> about IAM page etc in the
> > physical database chapter in Books Online). That is regardless whether you
> have a clustered index or
> > not.
> >
> > It doesn't have to be in the same sequence as the clustered index. The
> index can be heavy
> > fragmented, so the optimizer can decide to instead of jumping back and
> forth on the disk, it will
> > scan the file in physical order, according to the IAM page.
> >
> > --
> > Tibor Karaszi, SQL Server MVP
> > Archive at:
> http://groups.google.com/groups?oi=djq&as_ugroup=microsoft.public.sqlserver
> >
> >
> > "John" <spam@.spam.com> wrote in message
> news:OkuS1PW6DHA.1636@.TK2MSFTNGP12.phx.gbl...
> > > Hi,
> > >
> > > Is there any sort order if we don't use order by? for example,
> > >
> > > select * from tablename
> > >
> > > what is sort order? each time, I get the same records with the same
> order,
> > > any mystery?
> > >
> > >
> >
> >
>