SQL Server's DateTime
has the domain 1753-01-01 00:00:00.000 ≤ x ≤ 9999-12-31 23:59:59.997. The year 210 CE is outside that domain. Hence the problem.
If you were using SQL Server 2008 or later, you could cast it to a DateTime2
datatype and you'd be golden (its domain is 0001-01-01 00:00:00.0000000 &le x ≤ 9999-12-31 23:59:59.9999999. But with SQL Server 2005, you're pretty much SOL.
This is really a problem of data cleaning. My inclination in cases like this is to load the 3rd party data into a staging table with each field as character strings. Then cleanse the data in place, replacing, for instance, invalid dates with NULL. Once cleansed, then do the necessary conversion work to move it to its final destination.
Another approach is to use pattern matching and do the date filtering without converting anything to datetime
. ISO 8601 date/time values are character strings that have the laudable property of being (A) human-readable and (B) collating and comparing properly.
What I've done in the past is some analytical work to identify all the patterns in the datetime field by replacing decimal digits with a 'd' and then running group by
to compute the counts of each different pattern found. Once you have that you can create some pattern tables to guide you. Something like these:
create table #datePattern
(
pattern varchar(64) not null primary key clustered ,
monPos int not null ,
monLen int not null ,
dayPos int not null ,
dayLen int not null ,
yearPos int not null ,
yearLen int not null ,
)
insert #datePattern values ( '[0-9]/[0-9]/[0-9] %' ,1,1,3,1,5,1)
insert #datePattern values ( '[0-9]/[0-9]/[0-9][0-9] %' ,1,1,3,1,5,2)
insert #datePattern values ( '[0-9]/[0-9]/[0-9][0-9][0-9] %' ,1,1,3,1,5,3)
insert #datePattern values ( '[0-9]/[0-9]/[0-9][0-9][0-9][0-9] %' ,1,1,3,1,5,4)
insert #datePattern values ( '[0-9]/[0-9][0-9]/[0-9] %' ,1,1,3,2,6,1)
insert #datePattern values ( '[0-9]/[0-9][0-9]/[0-9][0-9] %' ,1,1,3,2,6,2)
insert #datePattern values ( '[0-9]/[0-9][0-9]/[0-9][0-9][0-9] %' ,1,1,3,2,6,3)
insert #datePattern values ( '[0-9]/[0-9][0-9]/[0-9][0-9][0-9][0-9] %' ,1,1,3,2,6,4)
insert #datePattern values ( '[0-9][0-9]/[0-9]/[0-9] %' ,1,2,4,1,6,1)
insert #datePattern values ( '[0-9][0-9]/[0-9]/[0-9][0-9] %' ,1,2,4,1,6,2)
insert #datePattern values ( '[0-9][0-9]/[0-9]/[0-9][0-9][0-9] %' ,1,2,4,1,6,3)
insert #datePattern values ( '[0-9][0-9]/[0-9]/[0-9][0-9][0-9][0-9] %' ,1,2,4,1,6,4)
insert #datePattern values ( '[0-9][0-9]/[0-9][0-9]/[0-9] %' ,1,2,4,2,7,1)
insert #datePattern values ( '[0-9][0-9]/[0-9][0-9]/[0-9][0-9] %' ,1,2,4,2,7,2)
insert #datePattern values ( '[0-9][0-9]/[0-9][0-9]/[0-9][0-9][0-9] %' ,1,2,4,2,7,3)
insert #datePattern values ( '[0-9][0-9]/[0-9][0-9]/[0-9][0-9][0-9][0-9] %' ,1,2,4,2,7,4)
create table #timePattern
(
pattern varchar(64) not null primary key clustered ,
hhPos int not null ,
hhLen int not null ,
mmPos int not null ,
mmLen int not null ,
ssPos int not null ,
ssLen int not null ,
)
insert #timePattern values ( '[0-9]:[0-9]:[0-9]' ,1,1,3,1,5,1 )
insert #timePattern values ( '[0-9]:[0-9]:[0-9][0-9]' ,1,1,3,1,5,2 )
insert #timePattern values ( '[0-9]:[0-9][0-9]:[0-9]' ,1,1,3,2,6,1 )
insert #timePattern values ( '[0-9]:[0-9][0-9]:[0-9][0-9]' ,1,1,3,2,6,2 )
insert #timePattern values ( '[0-9][0-9]:[0-9]:[0-9]' ,1,2,4,1,6,1 )
insert #timePattern values ( '[0-9][0-9]:[0-9]:[0-9][0-9]' ,1,2,4,1,6,2 )
insert #timePattern values ( '[0-9][0-9]:[0-9][0-9]:[0-9]' ,1,2,4,2,7,1 )
insert #timePattern values ( '[0-9][0-9]:[0-9][0-9]:[0-9][0-9]' ,1,2,4,2,7,2 )
You could combine these two tables into 1 but the number of combinations tends to explode things, though it greatly simplifies the query then.
Once you have that, the query is [fairly] easy, given that SQL is not exactly the world's best language choice for string processing:
---------------------------------------------------------------------
-- first, get your lower bound in ISO 8601 format yyyy-mm-dd hh:mm:ss
-- This will compare/collate properly
---------------------------------------------------------------------
declare @dtLowerBound varchar(255)
set @dtLowerBound = convert(varchar,dateadd(year,-1,current_timestamp),121)
-----------------------------------------------------------------
-- select rows with a start date more recent than the lower bound
-----------------------------------------------------------------
select isoDate = + right( '0000' + substring( t.startDate , coalesce(dt.yearPos,1) , coalesce(dt.YearLen,0) ) , 4 )
+ '-' + right( '00' + substring( t.startDate , coalesce(dt.monPos,1) , coalesce(dt.MonLen,0) ) , 2 )
+ '-' + right( '00' + substring( t.startDate , coalesce(dt.dayPos,1) , coalesce(dt.dayLen,0) ) , 2 )
+ case
when tm.pattern is not null then
' ' + right( '00' + substring(ltrim(rtrim( substring(t.startDate,dt.YearPos+dt.YearLen,1+len(t.startDate)-(dt.YearPos+dt.YearLen) ) ) ), tm.hhPos , tm.hhLen ) , 2 )
+ ':' + right( '00' + substring(ltrim(rtrim( substring(t.startDate,dt.YearPos+dt.YearLen,1+len(t.startDate)-(dt.YearPos+dt.YearLen) ) ) ), tm.mmPos , tm.mmLen ) , 2 )
+ ':' + right( '00' + substring(ltrim(rtrim( substring(t.startDate,dt.YearPos+dt.YearLen,1+len(t.startDate)-(dt.YearPos+dt.YearLen) ) ) ), tm.ssPos , tm.ssLen ) , 2 )
else ''
end
,*
from someTableWithBadData t
left join #datePattern dt on t.startDate like dt.pattern
left join #timePattern tm on ltrim(rtrim( substring(t.startDate,dt.YearPos+dt.YearLen,1+len(t.startDate)-(dt.YearPos+dt.YearLen) ) ) )
like tm.pattern
where @lowBound <= + right( '0000' + substring( t.startDate , coalesce(dt.yearPos,1) , coalesce(dt.YearLen,0) ) , 4 )
+ '-' + right( '00' + substring( t.startDate , coalesce(dt.monPos,1) , coalesce(dt.MonLen,0) ) , 2 )
+ '-' + right( '00' + substring( t.startDate , coalesce(dt.dayPos,1) , coalesce(dt.dayLen,0) ) , 2 )
+ case
when tm.pattern is not null then
' ' + right( '00' + substring(ltrim(rtrim( substring(t.startDate,dt.YearPos+dt.YearLen,1+len(t.startDate)-(dt.YearPos+dt.YearLen) ) ) ), tm.hhPos , tm.hhLen ) , 2 )
+ ':' + right( '00' + substring(ltrim(rtrim( substring(t.startDate,dt.YearPos+dt.YearLen,1+len(t.startDate)-(dt.YearPos+dt.YearLen) ) ) ), tm.mmPos , tm.mmLen ) , 2 )
+ ':' + right( '00' + substring(ltrim(rtrim( substring(t.startDate,dt.YearPos+dt.YearLen,1+len(t.startDate)-(dt.YearPos+dt.YearLen) ) ) ), tm.ssPos , tm.ssLen ) , 2 )
else ''
end
Like I said, SQL not the best choice for munging strings.
This should get you ... 90% there. Experience tells me that you'll still find more bad dates: months less than 1 or greater than 12 , days less than 1 or greater than 31, or days out of range for that month (nothing like February 31st to make the computer whine), etc. Old cobol programs in particular, loved to use a field of all 9s to indicate missing data, for instance (though that is an easy case to deal with).
My preferred technique is to write a perl script to scrub the data and bulk load it into SQL Server, using perl's BCP facilities. That's exactly the sort of problem space perl is designed for.