github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-07 20:31:06 -05:00
Code Issues Projects Releases Packages Wiki Activity
959 commits 3 branches 7 tags 201 MiB
Commit graph

388 commits

Author SHA1 Message Date
Henry Oswald
9a519f0d3d added docker runner into core codebase 
supports both local command runner and docker runner

added docker files for tex live

also fixed tests so they exit correctly & removed debug lines
 2018-03-14 15:44:49 +00:00
Henry Oswald
aa58b7a75f Merge branch 'ho-dockerise' of github.com:sharelatex/clsi-sharelatex into ho-dockerise 2018-03-02 18:31:09 +00:00
Henry Oswald
1320854da9 Merge branch 'master' into ho-dockerise 2018-03-02 18:16:16 +00:00
Henry Oswald
0589c8c544 Merge branch 'master' into ho-dockerise 2018-03-02 18:12:32 +00:00
Henry Oswald
f39d14bf1b unit tests pass, acceptence fail 
uncomment tests
 2018-03-02 17:34:41 +00:00
Henry Oswald
3399f55153 wip, docker container is correctly created 2018-03-02 17:14:23 +00:00
Henry Oswald
5cf3c904cb mvp 
needs hacked pacth in docker runner

wip

most tests pass
 2018-03-02 17:14:20 +00:00
Brian Gough
e18538bd38 log an error if core file is found in output 2018-01-03 15:41:31 +00:00
Brian Gough
3c937ce2d4 fix typo in log message 2017-10-02 15:45:09 +01:00
Brian Gough
392e96b81d move logging from SafeReader into caller 
prevent unnecessary logging when looking at headers of files where
hitting the end of the file is expected.
 2017-10-02 15:44:00 +01:00
Brian Gough
e209c48bd9 simplify tikzexternalize checks 2017-09-29 17:00:53 +01:00
Brian Gough
2bdba15bd5 keep tikzexternalize files 2017-09-29 16:02:23 +01:00
Brian Gough
ce51cc673f Merge branch 'bg-lock-compiles' 2017-09-28 13:16:29 +01:00
Brian Gough
a7cb7e6e4c use a separate function for hidden file check 2017-09-26 11:03:20 +01:00
Brian Gough
ef0db811e1 exclude hidden files from output 
express static server doesn't serve them and rejects with 404
 2017-09-26 10:42:59 +01:00
Brian Gough
2a23082c4e remove stat test for missing files 2017-09-26 09:48:09 +01:00
Brian Gough
d46943a7bb only exclude clsi-specific files from output list 2017-09-26 09:47:29 +01:00
Brian Gough
d464556f74 fix comment 2017-09-25 16:06:45 +01:00
Brian Gough
7f0e6f3eec lock compile directory 2017-09-22 16:19:33 +01:00
Brian Gough
81e8243827 fallback check for missing files 
dot files are not examined by OutputFileFinder, so do an extra check to
make sure those exist

also check for any relative paths in the resources
 2017-09-15 13:41:56 +01:00
Brian Gough
0a242bac3c rename saveProjectStateHash to saveProjectState 2017-09-08 13:56:40 +01:00
Brian Gough
66431fc2b8 log any missing files 2017-09-07 16:54:09 +01:00
Brian Gough
b8e8530a16 fix whitespace 2017-09-07 13:52:34 +01:00
Brian Gough
cbd3e32143 log error if state file is truncacted 2017-09-07 13:52:34 +01:00
Brian Gough
3d1c3a1d27 combine the resource state and resource list 
to prevent them getting out of sync
 2017-09-07 13:52:34 +01:00
Brian Gough
ea3ee82e74 added logging 2017-08-29 12:09:31 +01:00
Brian Gough
26f85ba75f read resource files safely 
put a limit on the amount of data read
 2017-08-18 11:17:01 +01:00
Brian Gough
a84c884fc9 splice state management into ResourceStateManager 2017-08-18 10:22:17 +01:00
Brian Gough
96b801b093 fix whitespace 2017-08-18 09:41:59 +01:00
Brian Gough
d3da4e1d3f ResourceWriter unit tests (wip) 2017-08-17 16:59:37 +01:00
Brian Gough
b901884248 avoid adding draft mode more than once 2017-08-17 15:03:37 +01:00
Brian Gough
8415ea2f71 store the resource list in a file 2017-08-17 14:53:35 +01:00
Brian Gough
2950c01130 add comment about syncType/syncState 2017-08-09 15:22:38 +01:00
Brian Gough
4ebc7e5e4a clean up the state file if no state passed in 2017-08-08 16:29:57 +01:00
Brian Gough
3a930a636e fix incremental request 2017-08-07 14:32:28 +01:00
Brian Gough
fe5ba5b619 restrict syncType values to full/incremental 2017-08-07 10:19:56 +01:00
Brian Gough
44ae011703 added files out of sync error object 2017-08-03 15:56:59 +01:00
Brian Gough
d4aad06c1f use syncType and syncState for clsi state options 2017-08-03 12:00:32 +01:00
Brian Gough
da86a094a8 write files incrementally 2017-08-02 13:46:10 +01:00
Brian Gough
fd5b557dde fix exception for empty content in TikzManager 2017-07-18 11:29:59 +01:00
Shane Kilkelly
577cf7ef4c Merge branch 'sk-reduce-kill-project-errors' 2017-06-27 10:03:51 +01:00
Shane Kilkelly
a55debb79f Send a 404 if the project files have gone away when running synctex. 
This is semantically nicer than the 500 response which used to be
produced in these circumstances.
 2017-06-23 14:46:40 +01:00
Shane Kilkelly
4b14de05c9 Killing an already stopped project is not an error 
Log a warning instead and continue.
 2017-06-20 09:18:15 +01:00
Brian Gough
f2746b7d5b delete intermediate xdv files from xelatex 2017-06-15 15:37:45 +01:00
Brian Gough
8d8f441ff8 Merge pull request #58 from sharelatex/bg-check-dir-before-synctex 
check file exists before running synctex
 2017-05-31 10:16:06 +01:00
Brian Gough
550979991f check directory exists and bail out on error 2017-05-31 10:06:27 +01:00
Brian Gough
42fa852b76 check file exists before running synctex 2017-05-24 10:09:43 +01:00
Brian Gough
7d93e77102 don't report compile timeouts to sentry 
just log them instead
 2017-05-24 09:42:05 +01:00
Brian Gough
86f29a4cfb add setting to avoid optimisations outside docker 2017-04-10 16:12:03 +01:00
Brian Gough
7e58bfd752 check if file is optimised before running qpdf 2017-04-04 16:50:06 +01:00
Brian Gough
4e34fa2fa6 Merge pull request #53 from sharelatex/bg-sanitise-paths 
additional check for valid rootResource
 2017-03-21 13:39:27 +00:00
Brian Gough
dd35c5d88b check for \tikzexternalize directly 
instead of \usepackage{tikz} and \usepackage{pgf}
 2017-03-21 11:36:08 +00:00
Brian Gough
1273a05ad4 fix path match 2017-03-21 11:30:32 +00:00
Brian Gough
54bdc8fed0 create separate function for path checking 2017-03-21 11:29:37 +00:00
Brian Gough
3bd919b3eb support for tikz externalize 
make copy of main file as output.tex for tikz externalize
 2017-03-20 10:55:28 +00:00
Brian Gough
4806da00a4 additional check for valid rootResource 2017-03-20 10:03:48 +00:00
Brian Gough
12b7a372e4 allow latexmk to pass through options 
this avoids problems in the latest version of latexmk where the
$pdflatex variable has been replaced by $xelatex and $lualatex when
running with -xelatex or -lualatex
 2017-03-02 16:43:35 +00:00
Brian Gough
8add9ff3ea fix exception in error log 2016-10-14 10:23:13 +01:00
Shane Kilkelly
34f47178fa Increase memory limit to 64mb 2016-09-28 11:02:58 +01:00
Shane Kilkelly
4efdddbbac Add CHKTEX_ULIMIT_OPTIONS 2016-09-23 15:32:37 +01:00
James Allen
638359e052 Don't delete knitr cache files 2016-09-22 14:14:29 +01:00
James Allen
4c04a5df3f Sanitize resource path along with rootResourcePath 2016-09-21 15:09:01 +01:00
Brian Gough
56429178df only run chktex on .tex files, not .Rtex files 
the .tex files produced from knitr have macros which confuse chktex
 2016-09-12 16:29:36 +01:00
Brian Gough
e23512f0e0 handle failed compile due to validation error 2016-08-24 15:46:47 +01:00
Brian Gough
481e842f33 fix commandRunner error to match dockerRunner 2016-08-24 15:45:26 +01:00
Brian Gough
1af44b32b3 remove chktex error 
too many false positives from 'unable to execute latex command'
 2016-08-22 15:11:39 +01:00
Brian Gough
ed5acc9f93 capture texcount error output 2016-08-11 10:26:08 +01:00
Brian Gough
400590b3ad read wordcount output asynchronously 2016-08-11 09:32:53 +01:00
Brian Gough
872dbc5215 add missing argument parameter to wordcount call 2016-08-11 09:29:03 +01:00
Brian Gough
c5917d4a75 use a command wrapper for synctex 
instead of an alternative child_process object
 2016-08-04 16:08:14 +01:00
Brian Gough
08a446f03d change logging message to be different from LatexRunner 2016-08-04 16:07:36 +01:00
Brian Gough
5c1e903b14 Merge pull request #44 from sharelatex/add-chktex-support 
Add chktex support
 2016-08-02 14:55:38 +01:00
Brian Gough
f0c7b455a5 provide setting to override child_process.execFile for synctex 2016-07-29 14:54:24 +01:00
Brian Gough
ffea361d12 provide validation mode where compilation always exits after chktex 2016-07-27 16:54:27 +01:00
Brian Gough
d6d6e18b89 run chktex when request has check:true 2016-07-26 16:22:38 +01:00
Brian Gough
3135ff2842 add support for passing additional environment parameters to command runner 
includes an example of passing environment variables to chktex
 2016-07-26 12:30:29 +01:00
Brian Gough
bd1e1de504 remove dead code 2016-07-18 11:05:45 +01:00
Brian Gough
9b58621fa8 add support for stopping compile 2016-07-14 16:43:52 +01:00
Brian Gough
dd4e1e17b5 keep one extra build until per-page pdf serving is enabled 2016-06-29 16:31:16 +01:00
Brian Gough
03188d2a57 only keep a single cached output directory in per-user containers 2016-06-28 09:28:40 +01:00
Shane Kilkelly
638ac52e40 Move the latexmk timing command into a configurable latexmkCommandPrefix. 
By default, no timing information will be taken.
On Linux with GNU user land, this value should be configured to `["/usr/bin/time", "-v"]`.
On Mac, gnu-time should be installed and configured to `["/usr/local/bin/gtime", "-v"]`.
 2016-06-17 14:38:08 +01:00
Brian Gough
d3395ef763 log user_id when clearing project 2016-06-02 15:32:33 +01:00
Brian Gough
5367bc22e5 add per-user routes for clearing cache and extend expiry methods 
this adds separate functionality for clearing the cache (assets and
database) and the project compile directory for a specific user
 2016-06-02 15:32:33 +01:00
Brian Gough
df641549c4 add per-user routes and methods 2016-06-02 15:32:31 +01:00
Brian Gough
1462e17f0c put the build id in the output file urls 
the url attribute will now give the preferred location for accessing
the output file, without the url having to be constructed by the web
client
 2016-06-02 15:30:50 +01:00
Henry Oswald
fdc8d83646 log out error on synctex 2016-05-27 16:18:18 +01:00
Henry Oswald
41ca3a4222 log out errors more clearly 2016-05-27 14:45:39 +01:00
Henry Oswald
448a30750b log out which command logger is used 2016-05-24 14:08:39 +01:00
Henry Oswald
ea8134c433 log out if the command running is being used 2016-05-23 15:45:39 +01:00
Henry Oswald
98a07e07ed parallelFileDownloads defaults to 1, sql can't take it 2016-05-23 14:31:27 +01:00
Henry Oswald
d140964c5f add error handler on CommandRunner 2016-05-23 14:13:55 +01:00
Brian Gough
340fef38a9 fix error in log for expiry timeout 2016-05-20 10:23:07 +01:00
Henry Oswald
786b9cdb84 Merge branch 'master' of https://github.com/sharelatex/clsi-sharelatex 2016-05-19 16:57:19 +01:00
Henry Oswald
9c873f99ba log out EXPIRY_TIMEOUT 2016-05-19 16:57:14 +01:00
Henry Oswald
9ae37d1814 make cached assets ttl set via config 2016-05-19 16:51:50 +01:00
Brian Gough
e286aede4e return the file path in the output file list for easy lookup 2016-05-19 16:38:18 +01:00
Brian Gough
2494327c12 make the build id a secure random token 
we allow existing build ids to work for backwards compatibility
this can be removed after some time
 2016-05-13 10:11:35 +01:00
Brian Gough
d5c1fb9997 fix logic excluding smoke test in metric 2016-05-10 10:10:01 +01:00
Brian Gough
00acfbdf86 fix tagname for graphite 2016-05-10 09:41:39 +01:00
Brian Gough
7779f65869 record system time 2016-05-10 09:12:13 +01:00
Brian Gough
d2e4253e1e preserve existing metric name 2016-05-10 09:12:00 +01:00
Brian Gough
3eebca4c01 add timing information from /usr/bin/time 2016-05-09 16:00:24 +01:00
Brian Gough
ebcf9a75ae add a metric for the TeXLive image used on each compile 2016-05-09 15:36:11 +01:00
Henry Oswald
7ab1654ebc move back to 2.5 days cache for moment 2016-04-21 17:40:09 +01:00
Henry Oswald
e024d3f503 added project status endpoint 
used for getting the server a project is on
 2016-04-20 15:38:05 +01:00
Henry Oswald
d416f75c2c increased clsi cache to 3.5 days 2016-04-13 09:29:57 +01:00
Brian Gough
27b739b081 don't log errors when files have disappeared from build directory 2016-04-07 16:16:39 +01:00
Brian Gough
4a785ff43c log errors in detail when file cannot be removed 2016-04-04 16:22:48 +01:00
Brian Gough
c1cb0f8800 suppress error when removing nonexistent file from cache 2016-03-31 13:33:42 +01:00
Brian Gough
01d7a59dfd remove console.log 2016-03-31 11:59:17 +01:00
Brian Gough
99c6b746f8 check directory exists before attempting to clear it 2016-03-31 11:59:17 +01:00
Brian Gough
26e94f2549 always create project directory when syncing resources to disk 
avoids errors when project is empty
 2016-03-31 11:59:17 +01:00
Brian Gough
308ca01b3c don't log missing files as warnings, but do report file access errors 2016-03-31 11:14:39 +01:00
James Allen
a2edeb800b Only archive main log and blg 2016-03-30 14:10:07 +01:00
James Allen
9c7e8af851 Ignore both .cache and .archive and other hidden files in finding output files 2016-03-30 11:41:11 +01:00
James Allen
66fa6a3e2e Don't timestamp strace logs otherwise it runs as anew container each time since the command changes 2016-03-30 10:59:01 +01:00
James Allen
da7b8b99fb Add in flags to run strace and capture logs 2016-03-30 10:37:22 +01:00
Brian Gough
285546fbcf bugfix - avoid double counting compiles 2016-03-17 14:37:34 +00:00
Brian Gough
456a8ca8cd add metric for qpdf 2016-03-17 09:55:18 +00:00
Brian Gough
a26f4fc5f6 add metrics for latexmk runs and errors 2016-03-17 09:55:18 +00:00
Henry Oswald
124c91e302 increased EXPIRY_TIMEOUT from 1.5 days to 2.5 days 2016-03-10 10:30:37 +00:00
James Allen
320e225700 Send .svg files as text/plain to prevent executable JS if they are loaded as SVG in the browser 2016-03-10 09:32:32 +00:00
James Allen
263ee43cb5 Make draft mode regex global 2016-02-02 15:28:59 +00:00
James Allen
97b5ac6a7f Remove left over debug log line 2016-02-02 14:28:51 +00:00
James Allen
0f55ce18ac Inject [draft] option to documentclass if draft option is passed 2016-02-02 14:26:14 +00:00
James Allen
37dbdc21bb Download up to 5 files in parallel 2016-02-01 13:19:16 +00:00
James Allen
de280d0ed4 Support configurable images in wordcount end point 2016-01-19 14:12:41 +00:00
James Allen
d924da0d2d Allow optional image name to be passed 2016-01-15 09:59:06 +00:00
Henry Oswald
e8e0b673eb fixed missing value in logger 2015-12-15 19:33:37 +00:00
Henry Oswald
253a99e780 added try catch around word count where a file is not created 2015-11-12 15:19:22 +00:00
Henry Oswald
069220dcc8 increased cache time to 1.5 days 2015-10-21 10:02:30 +01:00
Shane Kilkelly
3217b1d58f When serving output files, intelligently determine the appropriate content-type. 
cherry pick 6fa3fda3ed28239cf3ac9720629f9707663aa197 from datajoy.
 2015-09-21 16:59:35 +01:00
Henry Oswald
d7d0a6d9b0 Merge pull request #33 from sharelatex/pr/28 
wordcount
 2015-09-09 14:03:20 +01:00
Henry Oswald
4ee50b7239 - fixed bug with texcount returning wrong data for nauty lines 
- improved acceptence test for word count to use nauty lines
 2015-09-09 13:52:45 +01:00
Henry Oswald
ec338f8c10 add -inc to word count 
use -inc to word count included files

also moved private function to bottom
 2015-09-09 12:47:08 +01:00
Henry Oswald
d44cfb8614 added test to check compile should continue on error downloading http 
resource

also improved logging
 2015-09-09 09:44:38 +01:00
Henry Oswald
4b3aa933a3 don't error if a http resource can not be download 
try and continue, log the error but you might still be able to
compile. prevents issue with badly uploaded images in filstore
 2015-09-08 21:58:27 +01:00
Henrique Dias
d332665648 move texcount to docker 2015-09-08 10:19:46 -03:00
Henrique Dias
32c0a89ae1 add unit test 2015-06-08 19:27:47 -03:00
Henrique Dias
59e87a8729 initial version of texcount 2015-06-08 18:35:24 -03:00
Brian Gough
3dec42be2a Merge pull request #23 from sharelatex/add-mysql-indexes 
add indexes to mysql db
 2015-05-21 12:21:59 +01:00
Brian Gough
5b81a9b12e Merge pull request #22 from sharelatex/upgrade-sequelize 
upgrade sequelize and mysql
 2015-05-21 12:21:50 +01:00
Brian Gough
6ac7161f19 Merge pull request #21 from sharelatex/fix-urlfetcher-streams 
clean up error handling in UrlFetcher
 2015-05-21 12:21:31 +01:00
Brian Gough
44e1dc8c0c added comments 2015-05-21 11:33:13 +01:00
Brian Gough
e379068fa3 upgrade sequelize and mysql 2015-05-15 16:58:27 +01:00
Brian Gough
6ebcd9e8fd add indexes to db 2015-05-15 16:28:35 +01:00
Brian Gough
66fc2715dc clean up error handling in UrlFetcher 2015-05-15 16:28:11 +01:00
Brian Gough
608b1dd657 replace deprecated send(code,body) calls 2015-05-15 16:27:39 +01:00
Brian Gough
aa32cbc1ee clean up stream handling for file copy 2015-05-15 16:25:49 +01:00
Brian Gough
56f1ffa067 only run qpdf for the main output.pdf file 
was previously matching any pdf file, which caused it to run for
embedded pdf figures produced during the mklatex run
 2015-05-15 16:23:58 +01:00
Henry Oswald
7dfdad02f5 change regex checking file request ensure other files can not be accessed 2015-05-12 15:18:09 +01:00
Brian Gough
604c31ae84 additional validation of requests 2015-05-11 12:10:13 +01:00
Brian Gough
b08349e07c disable sequelize logging by default 
prevent any leaking of objects to console during debugging
 2015-05-05 10:03:17 +01:00
Brian Gough
76cd52af70 prevent leak of urlStream on failed downloads 2015-04-29 16:05:49 +01:00
Brian Gough
bb9944e618 log errors when downloading files and clean up failed downloads 2015-04-29 16:05:49 +01:00
Brian Gough
72918fb41b invalidate the cache if there is an error copying a file 2015-04-29 15:54:38 +01:00
Brian Gough
56b77c300c log errors when copying files from cache 2015-04-29 15:54:08 +01:00
Brian Gough
59c25fc703 remove unnecessary call to async.series in OutputFileFinder 
callback was previously async but is now synchronous, so high stack
usage.
 2015-03-16 16:47:25 +00:00
Brian Gough
3a3eda0ccf reduce cache limit for pdfs 2015-03-02 11:31:48 +00:00
Brian Gough
c26de9554d skip cache directory error when empty 2015-03-02 09:58:20 +00:00
Brian Gough
7f38c5e5a3 fix double callback for proc.on 'error' and proc.on 'close' 2015-02-27 16:07:02 +00:00
Brian Gough
e024fec82d provide a static server which forbids symlinks 
prevents mismatch between rootdir of server and rootdir of symlink
checking middleware
 2015-02-27 13:57:57 +00:00
Brian Gough
5fdd1d4f47 cleanup and logging 2015-02-27 13:15:35 +00:00
Brian Gough
a877dd36e7 remove debugging code 2015-02-26 15:32:01 +00:00
Brian Gough
5a5ef8baed accept build id parameter when serving static files 2015-02-25 17:05:19 +00:00
Brian Gough
49e1ce552d add an optimisation pass for the cached output files 2015-02-24 15:48:34 +00:00
Brian Gough
5c1d61c955 skip the cache directory when finding output files 2015-02-24 14:40:22 +00:00
Brian Gough
4002849e20 save output files in a .cache directory 2015-02-24 14:40:05 +00:00
Brian Gough
4532cd14b0 update sanitizePath regex 
remove accidental inclusion of , and add null char \x00
 2015-02-13 11:28:43 +00:00
James Allen
80382d5c15 Allow non-latin characters in the rootResourcePath 2015-02-13 11:21:35 +00:00
James Allen
561ce7dc60 Sanitize rootResourcePath 2015-02-11 16:39:43 +00:00
James Allen
1313b06fb7 Don't return error if directory doesn't exist yet 2014-12-09 11:25:23 +00:00
James Allen
42af7c4e63 Add in some debugging logging 2014-12-09 11:16:16 +00:00
James Allen
ffa7919e46 Use find -type f to get a list of output files 2014-12-09 11:08:07 +00:00
Henry Oswald
9742aa8fb4 replaced old symlink logic with tested middlewear based on fs.realpath 2014-12-04 23:54:22 +00:00
James Allen
49b7cdc854 Add in missing error check 2014-12-04 21:37:09 +00:00
Henry Oswald
c37447a2d6 increase max compile to 4 mins 2014-10-17 11:03:08 +01:00
Henry Oswald
3e4cfc5ba9 added url into error message when downloading from filestore 2014-06-10 14:09:36 +01:00
James Allen
fe6c76e62d Don't normalize path passed to synctex 2014-05-29 16:40:58 +01:00
James Allen
bc22a371d0 Explicitly look for output.pdf file, not any pdf 2014-05-29 15:49:48 +01:00
James Allen
5cd66aad58 Return more informative errors 2014-05-19 07:18:57 -04:00
James Allen
3213c562bc Add in new metrics 2014-05-09 14:55:37 +01:00
James Allen
52b22a41c8 Use simpler exec command with a timeout 2014-04-09 15:34:54 +01:00
James Allen
892e6bf6d1 Add in a synctex end point 2014-04-08 15:18:56 +01:00
James Allen
129cf0bf62 Add in support for markdown files 2014-04-04 09:56:20 -04:00
James Allen
b484f08d6e Use system rm -r to allow removal of files with broken char encodings 2014-04-02 12:53:02 +01:00
James Allen
73c652fb2c Update a missed setting 2014-02-13 11:08:40 +00:00
James Allen
46b36839e2 Initial open source commit 2014-02-12 17:27:43 +00:00