python - Dealing with large binary file (3 GB) in docker and Jenkins -


i'm using google model (binary file: around 3gb) in docker file , use jenkins build , deploy on production server. rest of code pulled bitbucket repo.

an example line docker file download , unzip file. happens once command cached. 

from python:2.7.13-onbuild  run mkdir -p /usr/src/app workdir /usr/src/app  arg debian_frontend=noninteractive run apt-get update && apt-get install --assume-yes apt-utils run apt-get update && apt-get install -y curl run apt-get update && apt-get install -y unzip run curl -o - https://s3.amazonaws.com/dl4j-distribution/googlenews-vectors-negative300.bin.gz \  | gunzip > /usr/src/app/googlenews-vectors-negative300.bin

everything works fine when build , run docker on local machine. however, when make patch-version push these changes production server through jenkins, build process fails @ end. setup, build , test phases works fine. however, post-build phase fails. (the build process pushes changes repo, , according logs, commands in docker file runs fine.) happens after , following error when looked @ logs.

18:49:27 654f45ecb7e3: layer exists 18:49:27 2c40c66f7667: layer exists 18:49:27 97108d083e01: pushed 18:49:31 35a4b123c0a3: pushed 18:50:10 1e730b4fb0a6: pushed 18:53:46 error parsing http 413 response body: invalid character '<' looking beginning of value: "<html>\r\n<head><title>413 request  `entity large</title></head>\r\n<body  bgcolor=\"white\">\r\n<center>`<h1>413 request entity large</h1></center>\r\n<hr> center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"

could file large?

before addition of file, docker , jenkins working fine too.

i wondering if there limitations in docker/jenkins in handling large file this? or breaking way approaching it.

update: increasing client_max_body_size solved specific error. however, getting error @ ssh -o stricthostkeychecking=no root@ipaddress "cd /root/ourapi &&docker-compose pull api &&docker-compose -p somefolder -d"

the docker-compose pull fails here unexpected eof. tries download image (1.6 gb) cancel after getting close size , retries ends wit eof error.

which brings me old question if large files needs handled different in situation?

update 2: the issue has been resolved. needed increase client_max_body_size 4 gb , need increase time-out parameter pulling repository our own repository server. adjusting these 2 parameters has led resolving problem.

the problem caused due following reasons:

  • the default value of client_max_body_size in ngnix server configuration low. due which, not upload file of 3.6 gb increased value 4 gb.
  • we run jetty server on our repository management system serve out http traffic needed increase time-out there jenkins pull relevant docker files there.

this answer in context of specific problem. however, question regarding how handle such files in better way still remains open. moreover, not clear if increasing client_max_body_size 4 gb idea in general.

relevant docs client_max_body_size: http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size


Comments

Post a Comment

Popular posts from this blog

php - Vagrant up error - Uncaught Reflection Exception: Class DOMDocument does not exist -

i've set bento/ubuntu-16.04 vagrant box using php 5.6, , getting following error quite few times when try , run shell script: "uncaught reflection exception: class domdocument not exist". i've tried adding following top of script (not together), each no success: "sudo apt-get install php5.6-xml", "sudo apt-get install php5-xml", "sudo apt-get install php-dom" i've tried resetting server "sudo service apache2 restart". does have suggestions?
Read more

vue.js - Create hooks for automated testing -

Image
qa guys complained can't automate frontend testing. because our html looks same outside. lazy hack did this <template> <div :role="$options.name"> ... </div> </template> <script> export default { name: 'vmcomponentname' } </script> which takes name script , applies html. in browser get: <div role="vmcomponentname"> ... </div> line :role="$options.name" goes in every component. is there dry er solutions? please share. additional details i'll explain in details looking for. imagine have vmusercreate form creates user. test case can create user . to automatically test without role, have use following selector: .wrapper > .wrapper > .wrapper > .submit-button the test extremely brittle (will break lot). if use roles can use [role="vmusercreateform"] .submit-button selector. many folds less brittle. so basically, i'm ...
Read more

Add new key value to json node in java -

i have json file : { "id":1, "name":"abc", "addressdetails": { "city":"newyork" } } i wanted add 1 more key-value ("pincode" : "414141") node 'addressdetails'. i tried using : objectmapper mapper = new objectmapper(); jsonnode root = mapper.readtree(new file("d://test.json")); objectnode node = mapper.createobjectnode(); node.with("addressdetails").put("pincode", "414141"); but it's not getting added, there way can ? you can try you should modify file path objectmapper mapper = new objectmapper(); objectnode nodes = mapper.readvalue(new file("d:\\test.txt"), objectnode.class); nodes.with("addressdetails").put("pincode", "414141"); mapper.writer().writevalue(new file("d:\\test.txt"), nodes);
Read more