mIRC Home    About    Download    Register    News    Help

Print Thread
$zip #264831 15/01/19 07:27 PM
Joined: Aug 2003
Posts: 236
P
Protopia Offline OP
Fjord artisan
OP Offline
Fjord artisan
P
Joined: Aug 2003
Posts: 236
$zip looks like a VERY useful enhancement, so thanks for adding it. However I am unclear what range of compressed file formats it supports (other than .zip)?

I have a particular requirement for reading .rar files as well as .zip.

How difficult would it be to convert this feature to use libarchive instead so that it would also include support for reading .rar and other formats as documented at https://github.com/libarchive/libarchive/wiki/LibarchiveFormats/ ?

Re: $zip [Re: Protopia] #264839 16/01/19 12:53 PM
Joined: Dec 2002
Posts: 4,521
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,521
$zip() only supports zip files, with or without AES, and no other formats.

I did try libarchive, unfortunately I just could not make it work. Everything from compilation issues to not being able to figure out how to use the API to zip files with AES. I eventually gave up :-] I may look at it again in future.

Re: $zip [Re: Khaled] #264852 18/01/19 12:27 AM
Joined: Feb 2003
Posts: 2,642
Raccoon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2003
Posts: 2,642
Would it be possible to add AES support to $encode/$decode() as well? I assume your crypto lib for SSL and now ZIP could have their functions repurposed.


Well. At least I won lunch.
Good philosophy, see good in bad, I like!
Re: $zip [Re: Raccoon] #264853 18/01/19 02:04 AM
Joined: Jan 2004
Posts: 1,178
maroon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Jan 2004
Posts: 1,178
A couple suggestions for $zip

1. The 'o' switch shouldn't be required when the export location exists as a foldername, but rather should be required only for overwriting actual files, which shouldn't include the dir-name entry inside the zip. This makes it impossible to unzip into an existing folder without using the 'o' switch, putting yourself at risk of overwriting an actual file. The error message in this case is confusing, as it's not invalid parameters just like it's not invalid parameters when /copy doesn't use -o when the output already exists.

2. Have a switch or .prop which reveals the contents of the zip to a script. Since $zip is possibly interacting with zips created with 7zip or other utilities, it's likely someone would receive a .zip via DCC which has unknown contents. This snippet is just a quick attempt to read the contents of a zip, so it won't work on all .zip files. For example it doesn't attempt to read the filesize of a zip64 format holding files whose original size is larger than 4gb, and it won't touch a zip which has a non-zero length comment at the very end of the file. It saves info to a hashtable, as well as creating a tab-delimited output as the /return value from calling this as an identifier.
Code:
; Syntax $ziplist(zipname,hashtablename)
; deletes then replaces hashtable, creates items 1+ containing. returns tab-delimited list of files
; hashtable format: counter_integer filesize|DIR filename
ziplist {
  var %size $file($1).size , %list , %ptr 1 , %counter 0 , %totsize 0
  if (%size !isnum 100-) { var %err Invalid zip | goto fail }
  bread $qt($1) $calc(%size -6) 6 &maroon.ziplist
  var %comlen $bvar(&maroon.ziplist,5).word , %cdirloc $bvar(&maroon.ziplist,1).long
  if (%comlen != 0) { var %err non-zero comment | goto fail }
  var %cdirsize $calc(%size - %cdirloc -6)
  if (%cdirsize !isnum 5-65536) { var %err invalid central dir | goto fail }
  bread $qt($1) $calc(%cdirloc) $calc(%cdirsize +7) &maroon.ziplist
  if ($bvar(&maroon.ziplist,0) != $calc(%cdirsize +6)) { var %err invalid zip | goto fail }
  while ($bvar(&maroon.ziplist,%ptr).long == 33639248) {
    var %modtime $base($bvar(&maroon.ziplist,$calc(%ptr +12)).word,10,16,4)
    var %moddate $base($bvar(&maroon.ziplist,$calc(%ptr +14)).word,10,16,4)
    var %crc     $base($bvar(&maroon.ziplist,$calc(%ptr +16)).long,10,16,8)
    var %arcsize $bvar(&maroon.ziplist,$calc(%ptr +20)).long
    var %orisize $bvar(&maroon.ziplist,$calc(%ptr +24)).long
    var %fnamlen $bvar(&maroon.ziplist,$calc(%ptr +28)).word
    var %xtralen $bvar(&maroon.ziplist,$calc(%ptr +30)).word
    var %comtlen $bvar(&maroon.ziplist,$calc(%ptr +32)).word
    var %intattr $bvar(&maroon.ziplist,$calc(%ptr +36)).word
    var %extattr $bvar(&maroon.ziplist,$calc(%ptr +38)).long
    inc %totsize %orisize
    inc %counter | if (%counter == 1) { hfree -w $2 | hmake $2 | echo -s $ $+ ziplist contents listing of $1 }
    hadd $2 %counter $iif($isbit(%extattr,5),DIR,%orisize) $bvar(&maroon.ziplist,$calc(%ptr + 46),%fnamlen).text
    if (!$isbit(%extattr,5)) var %list $addtok(%list,$bvar(&maroon.ziplist,$calc(%ptr + 46),%fnamlen).text,9)
    inc %ptr $calc(46+ %fnamlen + %xtralen + %comtlen)
  }
  if (%counter == 0) { var %err no contents found | goto fail }
  echo -sc info2 *ziplist %counter items for total size %totsize found in $1 and listed in hashtable $2
  return %list
  :fail | echo -sc info2 *$ziplist error: %err syntax: $ $+ ziplist(zipname,hashtable) | halt
}

Re: $zip [Re: Raccoon] #264855 18/01/19 09:11 AM
Joined: Dec 2002
Posts: 4,521
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,521
Quote:
Would it be possible to add AES support to $encode/$decode() as well? I assume your crypto lib for SSL and now ZIP could have their functions repurposed.

This is not possible as the AES support is built into the zip feature of the libzip package. It is not something that mIRC handles. I might be able to add AES support to $encode/$decode separately in future.

Re: $zip [Re: maroon] #264858 18/01/19 12:49 PM
Joined: Dec 2002
Posts: 4,521
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,521
Quote:
The 'o' switch shouldn't be required when the export location exists as a foldername

I personally want the current behaviour. If I am extracting a zip to an existing folder, it could overwrite files or change folder structure. So I chose to make $zip() fail if the file or folder exists, unless 'o' is used. That is the purpose of 'o' in this case. If you need a different behaviour, I am afraid you will need to script it.

As for the invalid parameter error, I will change this in the next version to report a file error.

Re: $zip [Re: Khaled] #264861 18/01/19 06:53 PM
Joined: Jan 2004
Posts: 1,178
maroon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Jan 2004
Posts: 1,178
$zip is failing to extract files if the filename in the central dir is different than in the local header, but that's probably a good thing.

But $zip should filter filenames containing ..

This trick doesn't seem to work when filename begins with leading slash, nor when filename begins with driveletter: But it is working when filename contains ..

file doesn't exist:
Code:
//echo -a $isfile(Scripts\script999.mrc)

create dummy file:
Code:
/write -c zzzzzzzzzzzzzzzzzzzzzzzzzzzz.mrc test

add it into a zip:
Code:
//remove test.zip | echo -a $zip(test.zip,c,zzzzzzzzzzzzzzzzzzzzzzzzzzzz.mrc)

alter filename inside zip:
Code:
//var %a test\..\..\Scripts\script999.mrc | bread test.zip 0 999 &v | bset -t &v 31 %a | bset -t &v 115 %a | bwrite -c test2.zip 0 999 &v

extract file from zip to non-existing foldername:
Code:
//rmdir nosuchfolder2 | echo -a $zip(test2.zip,e,nosuchfolder2) | echo -a $file(Scripts\script999.mrc).size

unzipped file now exists in scripts folder. now replace file with larger file, but this silently overwrites existing file, even creates file if Scripts\ subfolder doesn't exist:
Code:
//rmdir nosuchfolder2 | copy -o $qt(mirc.ini) scripts\script999.mrc | echo -a $file(Scripts\script999.mrc).size | echo -a $zip(test2.zip,e,nosuchfolder2) | echo -a $file(Scripts\script999.mrc).size

Re: $zip [Re: maroon] #264864 19/01/19 10:00 AM
Joined: Dec 2002
Posts: 4,521
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,521
Quote:
$zip is failing to extract files if the filename in the central dir is different than in the local header

This is due to how libzip handles this issue, so either way it would not be possible for mIRC to change it.

Quote:
But $zip should filter filenames containing ..

Ye gods. Thanks for spotting this! I never considered that the zip format would allow ".." in path names. I am not sure how best to handle this though. Should mIRC ignore specific files that contain ".." in their path but extract all others? Or should it assume that any zip file that contains ".." in a path name is malicious? I would opt to fail the entire zip file since, if it contains "..", it was most likely intended to be malicious and may contain other malicious files. This change will be in the next beta.

Re: $zip [Re: maroon] #264865 19/01/19 01:36 PM
Joined: Feb 2003
Posts: 2,642
Raccoon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2003
Posts: 2,642


Well. At least I won lunch.
Good philosophy, see good in bad, I like!
Re: $zip [Re: Raccoon] #264882 23/01/19 05:59 AM
Joined: Jan 2004
Posts: 1,178
maroon Offline
Hoopy frood
Offline
Hoopy frood
Joined: Jan 2004
Posts: 1,178
$zip l

Observing that filenames in the 'l' listing have the / slashes translated to \ format, but the subdir/ entry still has the slash as forward. Keeping a slash at the tail of a subdir entry is useful, identifying it as different from a zero size file.

Observing that the .size property for N=0 is being ignored, and would be useful if it returned total filesize instead of total item count.

This may be a limitation of the zip library, but it would be useful to have a parameter to alter the default compression level. For example specifying method=store when it's expected that the file can't be compressed, but is being zipped for either the password or for the crc32 verification of integrity.

Observing that the zip library isn't doing a fallback to storage when the file can't be compressed. This simplistic example creates a zip which compresses a 256-byte file to 261 bytes instead of storing as 256.

Code:
//var %size 256 | bset &v 1 $regsubex($str(x,%size),/x/g,$calc(\n -1) $chr(32)) | bwrite ascii.dat 0 %size &v | echo -a $zip(ascii.zip,c,ascii.dat)



Before $zip becomes established, it could be useful for the success to be the N number added to the zip instead of copying $compress which only compresses 1 file or not.

Conforming that 'e' is doing as you said, it's not extracting anything from a zip containing ../ or ..\. Not sure if it's intentional that in this case 'l' with N=0 reports 0. Not sure if 'l' should at least report filenames so people know why it failed.

Re: $zip [Re: maroon] #264883 23/01/19 10:00 AM
Joined: Dec 2002
Posts: 4,521
Khaled Offline
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 4,521
Quote:
Observing that filenames in the 'l' listing have the / slashes translated to \ format, but the subdir/ entry still has the slash as forward. Keeping a slash at the tail of a subdir entry is useful, identifying it as different from a zero size file.

This is actually what is being returned by libzip. mIRC is not making any changes to the results. I could make mIRC fix the slashes but then it wouldn't be what is in the zip file. On the other hand, not fixing the slashes means that scripts have to deal with both \ and / ... This change will be in the next beta.

Quote:
it would be useful to have a parameter to alter the default compression level

I would rather avoid adding more complexity. At today's internet speeds, I would generally opt for faster compression. Especially with $zip(), which does not support multiple CPU threads, etc., so it is going to be relatively slow anyway. I have set $zip() to use the fastest compression level by default.

Quote:
Not sure if it's intentional that in this case 'l' with N=0 reports 0

It is, the zip file is failed at the very earliest stage if it contains "..", so all zip-related features will see a failure. Another option is to handle this like 7zip; In some contexts, 7zip will convert ".." to underscores, in other contexts it will remove "..\" completely from the path (which can result in an empty file/folder name if all it contains is "..\..\..\"), and so on. I could make $zip() do all of this but if a zip file contains a zip exploit, I would rather just fail it consistently across all $zip() features.