I have a measurement program that pushes measurements into a web server, which is written in django and postgres. Currently I'm doing this update through a django form. What I've found, though, is that the form interface is pretty slow. If I set up a script, uploading 100 records through the form takes 7 or 8 seconds and pegs a cpu in my VM. Potentially I may end up with 20 or 30 measurement systems updating into the server, each running at 15 measures per second, so that update speed is not going to work.
I originally wrote it using the django ORM, then I rewrote it building the SQL string myself. Both equally slow. Then I commented out the database update entirely and its still pretty slow. So I guess the form interface is the bottleneck.
Questions:
- this is still in development so I'm still using "python manage.py runserver". will wsgi (or whatever) speed this up a bunch?
- what's the best practice for a programmatic single-record update into django?
here's the model for the form in question:
class MeasurementForm(forms.Form):
RunKey = forms.CharField()
MeasureTime = forms.CharField()
CaliperValue = forms.CharField()
CaliperToleranceStatus = forms.CharField()
IOFieldName = forms.CharField(required=False)
IOFieldValue = forms.CharField(required=False)
ToleranceStatus = forms.CharField()
Count = forms.CharField(required=False)
And here's the code for actually doing the update. The actual updating is commented out though.
def entermeasurement(request):
if request.method == 'POST': # If the form has been submitted...
form = MeasurementForm(request.POST) # A form bound to the POST data
response = HttpResponse(mimetype='text/plain')
if form.is_valid(): # All validation rules pass
raws = form.cleaned_data['CaliperValue'].split(",")
vals = []
for v in raws:
if "null" in unicode.lower(v):
vals.append(None)
else:
vals.append(v)
# insert a measurement record.
cursor = connection.cursor()
# print ("presql")
# get number of records removed given the date.
sqlstring = """insert into bwmeasures_measurements
("RunKey_id",
"MeasureTime",
"CaliperValue",
"CaliperToleranceStatus",
"IOFieldName",
"IOFieldValue",
"ToleranceStatus")
values
( """
"""
sqlstring = sqlstring + \
str(form.cleaned_data['RunKey']) + \
", timestamp'" + \
str(dateutil.parser.parse(form.cleaned_data['MeasureTime'])) + \
"', '" + \
postgresarray(vals) + \
"', '" + \
postgresarray(form.cleaned_data['CaliperToleranceStatus'].split(",")) + \
"', '" + \
postgresarray(quodstrings(form.cleaned_data['IOFieldName'])) + \
"', '" + \
postgresarray(quodstrings(form.cleaned_data['IOFieldValue'])) + \
"', '" + \
str(form.cleaned_data['ToleranceStatus']) + \
"')"
"""
# print(sqlstring)
#print("would update here!")
#cursor.execute(sqlstring)
#transaction.commit_unless_managed()
'''
# Django ORM way of doing the insert.
measurement = Measurements()
measurement.RunKey = ProductionRuns.objects.get(id=int(form.cleaned_data['RunKey']))
measurement.MeasureTime = dateutil.parser.parse(form.cleaned_data['MeasureTime'])
measurement.CaliperValue = vals
measurement.CaliperToleranceStatus = form.cleaned_data['CaliperToleranceStatus'].split(",")
measurement.IOFieldName = quodstrings(form.cleaned_data['IOFieldName'])
measurement.IOFieldValue = quodstrings(form.cleaned_data['IOFieldValue'])
measurement.ToleranceStatus = int(form.cleaned_data['ToleranceStatus'])
measurement.save()
'''
# indicate success
response.write("saved\n")
response.write(measurement.id)
# Process the data in form.cleaned_data
return response
else:
response.write("failed")
response.write(str(form.errors))
return response
else:
form = MeasurementForm() # An unbound form
return render(request, 'entermeasurement.html', {
'form': form,
})