1 # -*- coding: utf-8 -*- 2 # Copyright 2012 Google Inc. All Rights Reserved. 3 # 4 # Licensed under the Apache License, Version 2.0 (the "License"); 5 # you may not use this file except in compliance with the License. 6 # You may obtain a copy of the License at 7 # 8 # http://www.apache.org/licenses/LICENSE-2.0 9 # 10 # Unless required by applicable law or agreed to in writing, software 11 # distributed under the License is distributed on an "AS IS" BASIS, 12 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 # See the License for the specific language governing permissions and 14 # limitations under the License. 15 """Additional help about subdirectory handling in gsutil.""" 16 17 from __future__ import absolute_import 18 19 from gslib.help_provider import HelpProvider 20 21 _DETAILED_HELP_TEXT = (""" 22 <B>OVERVIEW</B> 23 This section provides details about how subdirectories work in gsutil. 24 Most users probably don't need to know these details, and can simply use 25 the commands (like cp -r) that work with subdirectories. We provide this 26 additional documentation to help users understand how gsutil handles 27 subdirectories differently than most GUI / web-based tools (e.g., why 28 those other tools create "dir_$folder$" objects), and also to explain cost and 29 performance implications of the gsutil approach, for those interested in such 30 details. 31 32 gsutil provides the illusion of a hierarchical file tree atop the "flat" 33 name space supported by the Google Cloud Storage service. To the service, 34 the object gs://your-bucket/abc/def/ghi.txt is just an object that happens to 35 have "/" characters in its name. There are no "abc" or "abc/def" directories; 36 just a single object with the given name. This diagram: 37 https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png 38 illustrates how gsutil provides a hierarchical view of objects in a bucket. 39 40 gsutil achieves the hierarchical file tree illusion by applying a variety of 41 rules, to try to make naming work the way users would expect. For example, in 42 order to determine whether to treat a destination URL as an object name or the 43 root of a directory under which objects should be copied gsutil uses these 44 rules: 45 46 1. If the destination object ends with a "/" gsutil treats it as a directory. 47 For example, if you run the command: 48 49 gsutil cp your-file gs://your-bucket/abc/ 50 51 gsutil will create the object gs://your-bucket/abc/your-file. 52 53 2. If the destination object is XYZ and an object exists called XYZ_$folder$ 54 gsutil treats XYZ as a directory. For example, if you run the command: 55 56 gsutil cp your-file gs://your-bucket/abc 57 58 and there exists an object called abc_$folder$, gsutil will create the 59 object gs://your-bucket/abc/your-file. 60 61 3. If you attempt to copy multiple source files to a destination URL, gsutil 62 treats the destination URL as a directory. For example, if you run 63 the command: 64 65 gsutil cp -r your-dir gs://your-bucket/abc 66 67 gsutil will create objects like gs://your-bucket/abc/your-dir/file1, etc. 68 (assuming file1 is a file under the source directory your-dir). 69 70 4. If none of the above rules applies, gsutil performs a bucket listing to 71 determine if the target of the operation is a prefix match to the 72 specified string. For example, if you run the command: 73 74 gsutil cp your-file gs://your-bucket/abc 75 76 gsutil will make a bucket listing request for the named bucket, using 77 delimiter="/" and prefix="abc". It will then examine the bucket listing 78 results and determine whether there are objects in the bucket whose path 79 starts with gs://your-bucket/abc/, to determine whether to treat the target 80 as an object name or a directory name. In turn this impacts the name of the 81 object you create: If the above check indicates there is an "abc" directory 82 you will end up with the object gs://your-bucket/abc/your-file; otherwise 83 you will end up with the object gs://your-bucket/abc. (See 84 "HOW NAMES ARE CONSTRUCTED" under "gsutil help cp" for more details.) 85 86 This rule-based approach stands in contrast to the way many tools work, which 87 create objects to mark the existence of folders (such as "dir_$folder$"). 88 gsutil understands several conventions used by such tools but does not 89 require such marker objects to implement naming behavior consistent with 90 UNIX commands. 91 92 A downside of the gsutil approach is it requires an extra bucket listing 93 before performing the needed cp or mv command. However those listings are 94 relatively inexpensive, because they use delimiter and prefix parameters to 95 limit result data. Moreover, gsutil makes only one bucket listing request 96 per cp/mv command, and thus amortizes the bucket listing cost across all 97 transferred objects (e.g., when performing a recursive copy of a directory 98 to the cloud). 99 """) 100 101 102 class CommandOptions(HelpProvider): 103 """Additional help about subdirectory handling in gsutil.""" 104 105 # Help specification. See help_provider.py for documentation. 106 help_spec = HelpProvider.HelpSpec( 107 help_name='subdirs', 108 help_name_aliases=[ 109 'dirs', 'directory', 'directories', 'folder', 'folders', 'hierarchy', 110 'subdir', 'subdirectory', 'subdirectories'], 111 help_type='additional_help', 112 help_one_line_summary='How Subdirectories Work', 113 help_text=_DETAILED_HELP_TEXT, 114 subcommand_help_text={}, 115 ) 116
